Gluster users - Nov 2018 - Q'apla brick does not come online with gluster 5.0, even with fresh install

Hi,

it occurs maybe the previous email was too many words and not enough 
data.  so will try to display the issue differently.

gluster created (single brick volume following advice from 
https://lists.gluster.org/pipermail/gluster-users/2016-October/028821.html):

root at sand1lian:~# gluster volume create moogle-gluster 
sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster

Gluster was started from cli with --debug, console reports the following 
with creation of the volume:

[2018-10-31 17:00:51.555918] D [MSGID: 0] 
[glusterd-volume-ops.c:328:__glusterd_handle_create_volume] 
0-management: Received create volume req
[2018-10-31 17:00:51.555963] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
[2018-10-31 17:00:51.556072] D [MSGID: 0] 
[glusterd-op-sm.c:209:glusterd_generate_txn_id] 0-management: 
Transaction_id = 3f5d14c9-ee08-493c-afac-d04d53c12aad
[2018-10-31 17:00:51.556090] D [MSGID: 0] 
[glusterd-op-sm.c:302:glusterd_set_txn_opinfo] 0-management: 
Successfully set opinfo for transaction ID : 
3f5d14c9-ee08-493c-afac-d04d53c12aad
[2018-10-31 17:00:51.556099] D [MSGID: 0] 
[glusterd-op-sm.c:309:glusterd_set_txn_opinfo] 0-management: Returning 0
[2018-10-31 17:00:51.556108] D [MSGID: 0] 
[glusterd-syncop.c:1809:gd_sync_task_begin] 0-management: Transaction ID 
: 3f5d14c9-ee08-493c-afac-d04d53c12aad
[2018-10-31 17:00:51.556127] D [MSGID: 0] 
[glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to 
acquire lock of vol moogle-gluster for 
bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
[2018-10-31 17:00:51.556293] D [MSGID: 0] 
[glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for vol 
moogle-gluster successfully held by bb8c61eb-f321-4485-8a8d-ddc369ac2203
[2018-10-31 17:00:51.556333] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
[2018-10-31 17:00:51.556368] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
[2018-10-31 17:00:51.556345] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
[2018-10-31 17:00:51.556368] D [MSGID: 0] 
[glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.556608] D [MSGID: 0] 
[glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
Returning 0
[2018-10-31 17:00:51.556656] D [MSGID: 0] 
[glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.556669] D [MSGID: 0] 
[store.c:473:gf_store_handle_destroy] 0-: Returning 0
[2018-10-31 17:00:51.556681] D [MSGID: 0] 
[glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
Returning 0
[2018-10-31 17:00:51.556690] D [MSGID: 0] 
[store.c:473:gf_store_handle_destroy] 0-: Returning 0
[2018-10-31 17:00:51.556699] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: 
Returning 0" repeated 3 times between [2018-10-31 17:00:51.556690] and 
[2018-10-31 17:00:51.556698]
[2018-10-31 17:00:51.556699] D [MSGID: 0] 
[glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
[2018-10-31 17:00:51.556728] D [MSGID: 0] 
[glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.556738] D [MSGID: 0] 
[glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
Returning 0
[2018-10-31 17:00:51.556752] D [MSGID: 0] 
[glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.556764] D [MSGID: 0] 
[store.c:473:gf_store_handle_destroy] 0-: Returning 0
[2018-10-31 17:00:51.556772] D [MSGID: 0] 
[glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
Returning 0
[2018-10-31 17:00:51.556781] D [MSGID: 0] 
[store.c:473:gf_store_handle_destroy] 0-: Returning 0
[2018-10-31 17:00:51.556791] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: 
Returning 0" repeated 3 times between [2018-10-31 17:00:51.556781] and 
[2018-10-31 17:00:51.556790]
[2018-10-31 17:00:51.556791] D [MSGID: 0] 
[glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
[2018-10-31 17:00:51.556818] D [MSGID: 0] 
[glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.556955] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.557033] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.557140] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.557154] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.557172] D [MSGID: 0] 
[glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
Returning 0
[2018-10-31 17:00:51.557183] D [MSGID: 0] 
[glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
[2018-10-31 17:00:51.557198] D [MSGID: 0] 
[glusterd-utils.c:7558:glusterd_new_brick_validate] 0-management: 
returning 0
[2018-10-31 17:00:51.557207] D [MSGID: 0] 
[glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
[2018-10-31 17:00:51.557392] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.557468] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.557542] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.557554] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.557573] D [MSGID: 0] 
[store.c:473:gf_store_handle_destroy] 0-: Returning 0
[2018-10-31 17:00:51.557586] D [MSGID: 0] 
[glusterd-volume-ops.c:1467:glusterd_op_stage_create_volume] 
0-management: Returning 0
[2018-10-31 17:00:51.557595] D [MSGID: 0] 
[glusterd-op-sm.c:6014:glusterd_op_stage_validate] 0-management: OP = 1. 
Returning 0
[2018-10-31 17:00:51.557610] D [MSGID: 0] 
[glusterd-op-sm.c:7659:glusterd_op_bricks_select] 0-management: Returning 0
[2018-10-31 17:00:51.557620] D [MSGID: 0] 
[glusterd-syncop.c:1751:gd_brick_op_phase] 0-management: Sent op req to 
0 bricks
[2018-10-31 17:00:51.557663] D [MSGID: 0] 
[glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.557693] D [MSGID: 0] 
[glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
[2018-10-31 17:00:51.557771] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.557844] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.557917] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.557931] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.557947] D [MSGID: 0] 
[glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
Returning 0
[2018-10-31 17:00:51.557957] D [MSGID: 0] 
[glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
[2018-10-31 17:00:51.558393] D [MSGID: 0] 
[xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning 0
[2018-10-31 17:00:51.558409] D [MSGID: 0] 
[glusterd-volgen.c:3140:_get_xlator_opt_key_from_vme] 0-glusterd: 
Returning 0
[2018-10-31 17:00:51.558495] W [MSGID: 101095] 
[xlator.c:180:xlator_volopt_dynload] 0-xlator: 
/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/nfs/server.so: cannot 
open shared object file: No such file or directory
[2018-10-31 17:00:51.558509] D [MSGID: 0] 
[xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning -1
[2018-10-31 17:00:51.558566] D [MSGID: 0] 
[glusterd-store.c:1107:glusterd_store_create_volume_dir] 0-management: 
Returning with 0
[2018-10-31 17:00:51.558593] D [MSGID: 0] 
[glusterd-store.c:1125:glusterd_store_create_volume_run_dir] 
0-management: Returning with 0
[2018-10-31 17:00:51.899586] D [MSGID: 0] 
[store.c:432:gf_store_handle_new] 0-: Returning 0
[2018-10-31 17:00:51.930562] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
[2018-10-31 17:00:51.930485] D [MSGID: 0] 
[store.c:432:gf_store_handle_new] 0-: Returning 0
[2018-10-31 17:00:51.930561] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.932563] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
0-management: returning: 0" repeated 19 times between [2018-10-31 
17:00:51.930561] and [2018-10-31 17:00:51.930794]
[2018-10-31 17:00:51.932562] D [MSGID: 0] 
[store.c:432:gf_store_handle_new] 0-: Returning 0
[2018-10-31 17:00:51.932688] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.932709] D [MSGID: 0] 
[glusterd-store.c:457:glusterd_store_snapd_write] 0-management: Returning 0
[2018-10-31 17:00:51.935196] D [MSGID: 0] 
[glusterd-store.c:521:glusterd_store_perform_snapd_store] 0-management: 
Returning 0
[2018-10-31 17:00:51.935226] D [MSGID: 0] 
[glusterd-store.c:585:glusterd_store_snapd_info] 0-management: Returning 
with 0
[2018-10-31 17:00:51.935251] D [MSGID: 0] 
[glusterd-store.c:788:_storeopts] 0-management: Storing in volinfo:key= 
transport.address-family, val=inet
[2018-10-31 17:00:51.935290] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.935314] D [MSGID: 0] 
[glusterd-store.c:788:_storeopts] 0-management: Storing in volinfo:key= 
nfs.disable, val=on
[2018-10-31 17:00:51.935344] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.935360] D [MSGID: 0] 
[glusterd-store.c:1174:glusterd_store_volinfo_write] 0-management: 
Returning 0
[2018-10-31 17:00:51.935382] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.936584] D [MSGID: 0] 
[store.c:432:gf_store_handle_new] 0-: Returning 0
[2018-10-31 17:00:51.936685] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.936807] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
0-management: returning: 0" repeated 10 times between [2018-10-31 
17:00:51.936685] and [2018-10-31 17:00:51.936806]
[2018-10-31 17:00:51.936807] D [MSGID: 0] 
[glusterd-store.c:430:glusterd_store_brickinfo_write] 0-management: 
Returning 0
[2018-10-31 17:00:51.936833] D [MSGID: 0] 
[glusterd-store.c:481:glusterd_store_perform_brick_store] 0-management: 
Returning 0
[2018-10-31 17:00:51.936841] D [MSGID: 0] 
[glusterd-store.c:550:glusterd_store_brickinfo] 0-management: Returning 
with 0
[2018-10-31 17:00:51.936848] D [MSGID: 0] 
[glusterd-store.c:1394:glusterd_store_brickinfos] 0-management: Returning 0
[2018-10-31 17:00:51.936856] D [MSGID: 0] 
[glusterd-store.c:1620:glusterd_store_perform_volume_store] 
0-management: Returning 0
[2018-10-31 17:00:51.958353] D [MSGID: 0] 
[store.c:386:gf_store_save_value] 0-management: returning: 0
[2018-10-31 17:00:51.958494] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
0-management: returning: 0" repeated 9 times between [2018-10-31 
17:00:51.958353] and [2018-10-31 17:00:51.958493]
[2018-10-31 17:00:51.958493] D [MSGID: 0] 
[glusterd-store.c:1558:glusterd_store_node_state_write] 0-management: 
Returning 0
[2018-10-31 17:00:51.960449] D [MSGID: 0] 
[glusterd-store.c:1592:glusterd_store_perform_node_state_store] 
0-management: Returning 0
[2018-10-31 17:00:51.960683] D [MSGID: 0] 
[glusterd-utils.c:2840:glusterd_volume_compute_cksum] 0-management: 
Returning with 0
[2018-10-31 17:00:51.960699] D [MSGID: 0] 
[glusterd-store.c:1832:glusterd_store_volinfo] 0-management: Returning 0
[2018-10-31 17:00:51.960797] D [MSGID: 0] 
[glusterd-utils.c:181:_brick_for_each] 0-management: Found a brick - 
sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
[2018-10-31 17:00:51.961200] D [MSGID: 0] 
[glusterd-volgen.c:1309:server_check_marker_off] 0-glusterd: Returning 0
[2018-10-31 17:00:51.961529] D [MSGID: 0] 
[glusterd-volgen.c:5816:generate_brick_volfiles] 0-management: Returning 0
[2018-10-31 17:00:51.961681] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.961756] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.961832] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.961846] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.961855] D [MSGID: 0] 
[glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
in volume moogle-gluster
[2018-10-31 17:00:51.961864] D [MSGID: 0] 
[glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
Returning 0
[2018-10-31 17:00:51.963126] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.963203] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.963280] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.963298] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.963308] D [MSGID: 0] 
[glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
in volume moogle-gluster
[2018-10-31 17:00:51.963316] D [MSGID: 0] 
[glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
Returning 0
[2018-10-31 17:00:51.964038] D [MSGID: 0] 
[glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
0-management: Unable to find friend: sand1lian.computerisms.ca
[2018-10-31 17:00:51.964112] D [MSGID: 0] 
[common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
[2018-10-31 17:00:51.964186] D [MSGID: 0] 
[common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
local address at interface eno1
[2018-10-31 17:00:51.964200] D [MSGID: 0] 
[glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
returning 0
[2018-10-31 17:00:51.964211] D [MSGID: 0] 
[glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
in volume moogle-gluster
[2018-10-31 17:00:51.964226] D [MSGID: 0] 
[glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
Returning 0
[2018-10-31 17:00:51.965159] D [MSGID: 0] 
[glusterd-op-sm.c:6150:glusterd_op_commit_perform] 0-management: Returning 0
[2018-10-31 17:00:51.965177] D [MSGID: 0] 
[glusterd-utils.c:9664:glusterd_aggr_brick_mount_dirs] 0-management: No 
brick_count present
[2018-10-31 17:00:51.965193] D [MSGID: 0] 
[glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx 
modification not required
[2018-10-31 17:00:51.965219] D [MSGID: 0] 
[glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to 
release lock of vol moogle-gluster for 
bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
[2018-10-31 17:00:51.966350] D [MSGID: 0] 
[glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for 
vol moogle-gluster successfully released
[2018-10-31 17:00:51.966462] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:00:51.966479] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:00:51.966509] D [MSGID: 0] 
[glusterd-op-sm.c:248:glusterd_get_txn_opinfo] 0-management: 
Successfully got opinfo for transaction ID : 
3f5d14c9-ee08-493c-afac-d04d53c12aad
[2018-10-31 17:00:51.966532] D [MSGID: 0] 
[glusterd-op-sm.c:252:glusterd_get_txn_opinfo] 0-management: Returning 0
[2018-10-31 17:00:51.966551] D [MSGID: 0] 
[glusterd-op-sm.c:352:glusterd_clear_txn_opinfo] 0-management: 
Successfully cleared opinfo for transaction ID : 
3f5d14c9-ee08-493c-afac-d04d53c12aad
[2018-10-31 17:00:51.966668] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
[2018-10-31 17:00:51.966561] D [MSGID: 0] 
[glusterd-op-sm.c:356:glusterd_clear_txn_opinfo] 0-management: Returning 0
[2018-10-31 17:00:51.966667] D [MSGID: 0] 
[glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
Returning 0
[2018-10-31 17:00:51.968134] D [socket.c:2927:socket_event_handler] 
0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
[2018-10-31 17:00:51.968183] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler
grep: /var/lib/glusterd/vols/moogle-gluster/bricks/*: No such file or 
directory
[2018-10-31 17:00:51.975661] I [run.c:242:runner_log] 
(-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe0dbe)
[0x7f3f248dbdbe] 
-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe07fe) 
[0x7f3f248db7fe] 
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) 
[0x7f3f28ac35a5] ) 0-management: Ran script: 
/var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh 
--volname=moogle-gluster
[2018-10-31 17:01:12.466614] D [logging.c:1871:gf_log_flush_timeout_cbk] 
0-logging-infra: Log timer timed out. About to flush outstanding 
messages if present
[2018-10-31 17:01:12.466667] D 
[logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
timer now. Timeout = 120, current buf size = 5
[2018-10-31 17:03:12.492414] D [logging.c:1871:gf_log_flush_timeout_cbk] 
0-logging-infra: Log timer timed out. About to flush outstanding 
messages if present
[2018-10-31 17:03:12.492447] D 
[logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
timer now. Timeout = 120, current buf size = 5

Not sure about the unable to find friend message:

root at sand1lian:~# dig +short sand1lian.computerisms.ca
192.168.25.52

start the volume:

root at sand1lian:~# gluster v start moogle-gluster
volume start: moogle-gluster: failed: Commit failed on localhost. Please 
check log file for details.

output of cli.log while issuing start command:

[2018-10-31 17:08:49.019079] I [cli.c:764:main] 0-cli: Started running 
gluster with version 5.0
[2018-10-31 17:08:49.021694] W [socket.c:3365:socket_connect] 
0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
supported"
[2018-10-31 17:08:49.021924] W [socket.c:3365:socket_connect] 
0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
supported"
[2018-10-31 17:08:49.101120] I [MSGID: 101190] 
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2018-10-31 17:08:49.101231] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler
[2018-10-31 17:08:49.113485] I 
[cli-rpc-ops.c:1419:gf_cli_start_volume_cbk] 0-cli: Received resp to 
start volume
[2018-10-31 17:08:49.113626] I [input.c:31:cli_batch] 0-: Exiting with: -1

and output of brick log while starting volume:

[2018-10-31 17:08:49.107966] I [MSGID: 100030] [glusterfsd.c:2691:main] 
0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 5.0 
(args: /usr/sbin/glusterfsd -s sand1lian.computerisms.ca --volfile-id 
moogle-gluster.sand1lian.computerisms.ca.var-GlusterBrick-moogle-gluster 
-p 
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
-S /var/run/gluster/f41bfcfaf40deb7d.socket --brick-name 
/var/GlusterBrick/moogle-gluster -l 
/var/log/glusterfs/bricks/var-GlusterBrick-moogle-gluster.log 
--xlator-option 
*-posix.glusterd-uuid=bb8c61eb-f321-4485-8a8d-ddc369ac2203 
--process-name brick --brick-port 49157 --xlator-option 
moogle-gluster-server.listen-port=49157)
[2018-10-31 17:08:49.112123] E [socket.c:3466:socket_connect] 
0-glusterfs: connection attempt on  failed, (Invalid argument)
[2018-10-31 17:08:49.112293] I [MSGID: 101190] 
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2018-10-31 17:08:49.112374] I [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 
0-glusterfsd-mgmt: disconnected from remote-host: sand1lian.computerisms.ca
[2018-10-31 17:08:49.112399] I [glusterfsd-mgmt.c:2444:mgmt_rpc_notify] 
0-glusterfsd-mgmt: Exhausted all volfile servers
[2018-10-31 17:08:49.112656] W [glusterfsd.c:1481:cleanup_and_exit] 
(-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xf023) [0x7f3466c12023] 
-->/usr/sbin/glusterfsd(+0x1273e) [0x557f4ea6373e] 
-->/usr/sbin/glusterfsd(cleanup_and_exit+0x54) [0x557f4ea5be94] ) 0-: 
received signum (1), shutting down
[2018-10-31 17:08:49.112973] E [socket.c:3466:socket_connect] 
0-glusterfs: connection attempt on  failed, (Invalid argument)
[2018-10-31 17:08:49.112996] W [rpc-clnt.c:1683:rpc_clnt_submit] 
0-glusterfs: error returned while attempting to connect to host:(null), 
port:0
[2018-10-31 17:08:49.113007] I 
[socket.c:3710:socket_submit_outgoing_msg] 0-glusterfs: not connected 
(priv->connected = 0)
[2018-10-31 17:08:49.113016] W [rpc-clnt.c:1695:rpc_clnt_submit] 
0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x2 Program: 
Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)


still seeing the empty pid file and the connection attempt on  failed, 
(Invalid argument) as the mostly likely culprits, but have read 
everything of relevance I have found on google and not discovered a 
solution yet...















On 2018-10-30 9:15 p.m., Computerisms Corporation wrote:
> Hi,
> 
> Fortunately I am playing in a sandbox right now, but I am good and stuck 
> and hoping someone can point me in the right direction.
> 
> I have been playing for about 3 months with a gluster that currently has 
> one brick.? The idea is that I have a server with data, I need to 
> migrate that server onto the new gluster-capable server, then I can use 
> the original server to make a 2nd brick, then I will be able to make 
> some room on a 3rd server for an arbiter brick.? So I am building and 
> testing to be sure it all works before I try it in production.
> 
> Yesterday morning I was plugging away at figuring out how to make stuff 
> work on the new gluster server when I ran into an issue trying to rm -rf 
> a directory and it telling me it wasn't empty when ls -al showed that
it
> was.? This has happened to me before, and what I did to fix it before 
> was unmount the Glusterfs, go into the brick, delete the files, and 
> remount the Glusterfs.? I did that and it appeared to mount fine, but 
> when I tried to access the gluster mount, it gave me an error that there 
> were too many levels of symlinks.
> 
> I spent my day yesterday trying pretty much everything I could find on 
> google and a few things I couldn't.? In the past when stuff has gone 
> funny with gluster on this box, I have always shut everything down and 
> checked if there was a new version of gluster, and indeed there was 
> version 5.0 available.? So I did the upgrade quite early in the day. 
> Sadly it didn't fix my problem, but it did give me an error that led me
> to modifying my hosts file to be ipv6 resolvable.? Also after that, the 
> only time the gluster would mount was at reboot, but always with the 
> symlinks error, and it wasn't really mounted as reported by mount, but 
> the directory could be unmounted.
> 
> Having struck out completely yesterday, today I decided to start with a 
> new machine.? I kept a history of the commands I had used to build the 
> gluster a few months back and pasted them all in.? Found that the 5.0 
> package does not enable systemd, found that I needed the ipv6 entries in 
> the hosts file again, and also found the same problem in that the 
> glusterfs would not mount, the symlinks error at reboot, and the same 
> log entries.
> 
> I am still pretty new with gluster, so my best may not be that good, but 
> as best as I can tell the issue is that the brick will not start, even 
> with the force option.? I think the problem boils down to one or both of 
> two lines in the logs.? In the glusterd.log I have a line:
> 
> 0-: Unable to read pidfile: 
>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
> 
> 
> The file exists, and I can't see anything wrong with permissions on the
> file or the file tree leading to it, but it is a zero-bit file, so I am 
> thinking the problem is not the file, but that it can't read the 
> contents of the file because there aren't any.
> 
> The other log entry is in the brick log:
> 
> 0-glusterfs: connection attempt on? failed, (Invalid argument)
> 
> When I looked this up, it seems in my case there should be an attempt to 
> connect on 127.0.0.1, but given the double space I am thinking the host 
> argument is null, hence the invalid argument.? It occurs that maybe I 
> still need some other entry in my hosts file to satisfy this, but I 
> can't think what it would be.? I have created DNS entries; dig works, 
> and both hostname and FQDN resolve.
> 
> I have tried to change a lot of things today, so probably things are 
> buggered up beyond hope right now so even if I do find the solution 
> maybe it won't work.? will wipe the new machine and start over again 
> tomorrow.
> 
> I realize the post is kinda long, sorry for that, but I want to make 
> sure I get every thing important.? In fairness, though, I could easily 
> double the length of this post with possibly relevant things (if you are 
> interested).? If you are still reading, thank you so much, I would 
> appreciate anything, even a wild guess, as to how to move forward on this?
>

forgot to add output of glusterd console when starting the volume:

[2018-10-31 17:31:33.887923] D [MSGID: 0] 
[glusterd-volume-ops.c:572:__glusterd_handle_cli_start_volume] 
0-management: Received start vol req for volume moogle-gluster
[2018-10-31 17:31:33.887976] D [MSGID: 0] 
[glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to 
acquire lock of vol moogle-gluster for 
bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
[2018-10-31 17:31:33.888171] D [MSGID: 0] 
[glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for vol 
moogle-gluster successfully held by bb8c61eb-f321-4485-8a8d-ddc369ac2203
[2018-10-31 17:31:33.888189] D [MSGID: 0] 
[glusterd-locks.c:519:glusterd_multiple_mgmt_v3_lock] 0-management: 
Returning 0
[2018-10-31 17:31:33.888204] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.888213] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.888229] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.888237] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.888247] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.888256] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.888269] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.888277] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.888294] D [MSGID: 0] 
[glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
[2018-10-31 17:31:33.888318] D [MSGID: 0] 
[glusterd-mgmt.c:223:gd_mgmt_v3_pre_validate_fn] 0-management: OP = 5. 
Returning 0
[2018-10-31 17:31:33.888668] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.888682] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.888719] E [MSGID: 101012] 
[common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile: 
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
[2018-10-31 17:31:33.888757] I 
[glusterd-utils.c:6300:glusterd_brick_start] 0-management: starting a 
fresh brick process for brick /var/GlusterBrick/moogle-gluster
[2018-10-31 17:31:33.898943] D [logging.c:1998:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
[2018-10-31 17:31:33.888780] E [MSGID: 101012] 
[common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile: 
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
[2018-10-31 17:31:33.898942] E [MSGID: 106005] 
[glusterd-utils.c:6305:glusterd_brick_start] 0-management: Unable to 
start brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
[2018-10-31 17:31:33.899068] D [MSGID: 0] 
[glusterd-utils.c:6315:glusterd_brick_start] 0-management: returning -107
[2018-10-31 17:31:33.899088] E [MSGID: 106122] 
[glusterd-mgmt.c:308:gd_mgmt_v3_commit_fn] 0-management: Volume start 
commit failed.
[2018-10-31 17:31:33.899100] D [MSGID: 0] 
[glusterd-mgmt.c:392:gd_mgmt_v3_commit_fn] 0-management: OP = 5. 
Returning -107
[2018-10-31 17:31:33.899114] E [MSGID: 106122] 
[glusterd-mgmt.c:1557:glusterd_mgmt_v3_commit] 0-management: Commit 
failed for operation Start on local node
[2018-10-31 17:31:33.899128] D [MSGID: 0] 
[glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx 
modification not required
[2018-10-31 17:31:33.899140] E [MSGID: 106122] 
[glusterd-mgmt.c:2160:glusterd_mgmt_v3_initiate_all_phases] 
0-management: Commit Op Failed
[2018-10-31 17:31:33.899168] D [MSGID: 0] 
[glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to 
release lock of vol moogle-gluster for 
bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
[2018-10-31 17:31:33.899195] D [MSGID: 0] 
[glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for 
vol moogle-gluster successfully released
[2018-10-31 17:31:33.899211] D [MSGID: 0] 
[glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
moogle-gluster found
[2018-10-31 17:31:33.899221] D [MSGID: 0] 
[glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
[2018-10-31 17:31:33.899232] D [MSGID: 0] 
[glusterd-locks.c:464:glusterd_multiple_mgmt_v3_unlock] 0-management: 
Returning 0
[2018-10-31 17:31:33.899314] D [MSGID: 0] 
[glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
Returning 0
[2018-10-31 17:31:33.900750] D [socket.c:2927:socket_event_handler] 
0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
[2018-10-31 17:31:33.900809] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler


On 2018-10-31 10:19 a.m., Computerisms Corporation
wrote:
> Hi,
> 
> it occurs maybe the previous email was too many words and not enough 
> data.? so will try to display the issue differently.
> 
> gluster created (single brick volume following advice from 
>
https://lists.gluster.org/pipermail/gluster-users/2016-October/028821.html):
> 
> 
> root at sand1lian:~# gluster volume create moogle-gluster 
> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
> 
> Gluster was started from cli with --debug, console reports the following 
> with creation of the volume:
> 
> [2018-10-31 17:00:51.555918] D [MSGID: 0] 
> [glusterd-volume-ops.c:328:__glusterd_handle_create_volume] 
> 0-management: Received create volume req
> [2018-10-31 17:00:51.555963] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
> [2018-10-31 17:00:51.556072] D [MSGID: 0] 
> [glusterd-op-sm.c:209:glusterd_generate_txn_id] 0-management: 
> Transaction_id = 3f5d14c9-ee08-493c-afac-d04d53c12aad
> [2018-10-31 17:00:51.556090] D [MSGID: 0] 
> [glusterd-op-sm.c:302:glusterd_set_txn_opinfo] 0-management: 
> Successfully set opinfo for transaction ID : 
> 3f5d14c9-ee08-493c-afac-d04d53c12aad
> [2018-10-31 17:00:51.556099] D [MSGID: 0] 
> [glusterd-op-sm.c:309:glusterd_set_txn_opinfo] 0-management: Returning 0
> [2018-10-31 17:00:51.556108] D [MSGID: 0] 
> [glusterd-syncop.c:1809:gd_sync_task_begin] 0-management: Transaction ID 
> : 3f5d14c9-ee08-493c-afac-d04d53c12aad
> [2018-10-31 17:00:51.556127] D [MSGID: 0] 
> [glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to 
> acquire lock of vol moogle-gluster for 
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:00:51.556293] D [MSGID: 0] 
> [glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for vol 
> moogle-gluster successfully held by bb8c61eb-f321-4485-8a8d-ddc369ac2203
> [2018-10-31 17:00:51.556333] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
> [2018-10-31 17:00:51.556368] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> [2018-10-31 17:00:51.556345] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
> [2018-10-31 17:00:51.556368] D [MSGID: 0] 
> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.556608] D [MSGID: 0] 
> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.556656] D [MSGID: 0] 
> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.556669] D [MSGID: 0] 
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [2018-10-31 17:00:51.556681] D [MSGID: 0] 
> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.556690] D [MSGID: 0] 
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [2018-10-31 17:00:51.556699] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: 
> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556690] and
> [2018-10-31 17:00:51.556698]
> [2018-10-31 17:00:51.556699] D [MSGID: 0] 
> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
> [2018-10-31 17:00:51.556728] D [MSGID: 0] 
> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.556738] D [MSGID: 0] 
> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.556752] D [MSGID: 0] 
> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.556764] D [MSGID: 0] 
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [2018-10-31 17:00:51.556772] D [MSGID: 0] 
> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.556781] D [MSGID: 0] 
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [2018-10-31 17:00:51.556791] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: 
> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556781] and
> [2018-10-31 17:00:51.556790]
> [2018-10-31 17:00:51.556791] D [MSGID: 0] 
> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
> [2018-10-31 17:00:51.556818] D [MSGID: 0] 
> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.556955] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.557033] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.557140] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.557154] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.557172] D [MSGID: 0] 
> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.557183] D [MSGID: 0] 
> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
> [2018-10-31 17:00:51.557198] D [MSGID: 0] 
> [glusterd-utils.c:7558:glusterd_new_brick_validate] 0-management: 
> returning 0
> [2018-10-31 17:00:51.557207] D [MSGID: 0] 
> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
> [2018-10-31 17:00:51.557392] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.557468] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.557542] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.557554] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.557573] D [MSGID: 0] 
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [2018-10-31 17:00:51.557586] D [MSGID: 0] 
> [glusterd-volume-ops.c:1467:glusterd_op_stage_create_volume] 
> 0-management: Returning 0
> [2018-10-31 17:00:51.557595] D [MSGID: 0] 
> [glusterd-op-sm.c:6014:glusterd_op_stage_validate] 0-management: OP = 1. 
> Returning 0
> [2018-10-31 17:00:51.557610] D [MSGID: 0] 
> [glusterd-op-sm.c:7659:glusterd_op_bricks_select] 0-management: Returning 0
> [2018-10-31 17:00:51.557620] D [MSGID: 0] 
> [glusterd-syncop.c:1751:gd_brick_op_phase] 0-management: Sent op req to 
> 0 bricks
> [2018-10-31 17:00:51.557663] D [MSGID: 0] 
> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.557693] D [MSGID: 0] 
> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
> [2018-10-31 17:00:51.557771] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.557844] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.557917] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.557931] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.557947] D [MSGID: 0] 
> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.557957] D [MSGID: 0] 
> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
> [2018-10-31 17:00:51.558393] D [MSGID: 0] 
> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning 0
> [2018-10-31 17:00:51.558409] D [MSGID: 0] 
> [glusterd-volgen.c:3140:_get_xlator_opt_key_from_vme] 0-glusterd: 
> Returning 0
> [2018-10-31 17:00:51.558495] W [MSGID: 101095] 
> [xlator.c:180:xlator_volopt_dynload] 0-xlator: 
> /usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/nfs/server.so: cannot 
> open shared object file: No such file or directory
> [2018-10-31 17:00:51.558509] D [MSGID: 0] 
> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning -1
> [2018-10-31 17:00:51.558566] D [MSGID: 0] 
> [glusterd-store.c:1107:glusterd_store_create_volume_dir] 0-management: 
> Returning with 0
> [2018-10-31 17:00:51.558593] D [MSGID: 0] 
> [glusterd-store.c:1125:glusterd_store_create_volume_run_dir] 
> 0-management: Returning with 0
> [2018-10-31 17:00:51.899586] D [MSGID: 0] 
> [store.c:432:gf_store_handle_new] 0-: Returning 0
> [2018-10-31 17:00:51.930562] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> [2018-10-31 17:00:51.930485] D [MSGID: 0] 
> [store.c:432:gf_store_handle_new] 0-: Returning 0
> [2018-10-31 17:00:51.930561] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.932563] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
> 0-management: returning: 0" repeated 19 times between [2018-10-31 
> 17:00:51.930561] and [2018-10-31 17:00:51.930794]
> [2018-10-31 17:00:51.932562] D [MSGID: 0] 
> [store.c:432:gf_store_handle_new] 0-: Returning 0
> [2018-10-31 17:00:51.932688] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.932709] D [MSGID: 0] 
> [glusterd-store.c:457:glusterd_store_snapd_write] 0-management: Returning 0
> [2018-10-31 17:00:51.935196] D [MSGID: 0] 
> [glusterd-store.c:521:glusterd_store_perform_snapd_store] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.935226] D [MSGID: 0] 
> [glusterd-store.c:585:glusterd_store_snapd_info] 0-management: Returning 
> with 0
> [2018-10-31 17:00:51.935251] D [MSGID: 0] 
> [glusterd-store.c:788:_storeopts] 0-management: Storing in volinfo:key= 
> transport.address-family, val=inet
> [2018-10-31 17:00:51.935290] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.935314] D [MSGID: 0] 
> [glusterd-store.c:788:_storeopts] 0-management: Storing in volinfo:key= 
> nfs.disable, val=on
> [2018-10-31 17:00:51.935344] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.935360] D [MSGID: 0] 
> [glusterd-store.c:1174:glusterd_store_volinfo_write] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.935382] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.936584] D [MSGID: 0] 
> [store.c:432:gf_store_handle_new] 0-: Returning 0
> [2018-10-31 17:00:51.936685] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.936807] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
> 0-management: returning: 0" repeated 10 times between [2018-10-31 
> 17:00:51.936685] and [2018-10-31 17:00:51.936806]
> [2018-10-31 17:00:51.936807] D [MSGID: 0] 
> [glusterd-store.c:430:glusterd_store_brickinfo_write] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.936833] D [MSGID: 0] 
> [glusterd-store.c:481:glusterd_store_perform_brick_store] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.936841] D [MSGID: 0] 
> [glusterd-store.c:550:glusterd_store_brickinfo] 0-management: Returning 
> with 0
> [2018-10-31 17:00:51.936848] D [MSGID: 0] 
> [glusterd-store.c:1394:glusterd_store_brickinfos] 0-management: Returning 0
> [2018-10-31 17:00:51.936856] D [MSGID: 0] 
> [glusterd-store.c:1620:glusterd_store_perform_volume_store] 
> 0-management: Returning 0
> [2018-10-31 17:00:51.958353] D [MSGID: 0] 
> [store.c:386:gf_store_save_value] 0-management: returning: 0
> [2018-10-31 17:00:51.958494] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
> 0-management: returning: 0" repeated 9 times between [2018-10-31 
> 17:00:51.958353] and [2018-10-31 17:00:51.958493]
> [2018-10-31 17:00:51.958493] D [MSGID: 0] 
> [glusterd-store.c:1558:glusterd_store_node_state_write] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.960449] D [MSGID: 0] 
> [glusterd-store.c:1592:glusterd_store_perform_node_state_store] 
> 0-management: Returning 0
> [2018-10-31 17:00:51.960683] D [MSGID: 0] 
> [glusterd-utils.c:2840:glusterd_volume_compute_cksum] 0-management: 
> Returning with 0
> [2018-10-31 17:00:51.960699] D [MSGID: 0] 
> [glusterd-store.c:1832:glusterd_store_volinfo] 0-management: Returning 0
> [2018-10-31 17:00:51.960797] D [MSGID: 0] 
> [glusterd-utils.c:181:_brick_for_each] 0-management: Found a brick - 
> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
> [2018-10-31 17:00:51.961200] D [MSGID: 0] 
> [glusterd-volgen.c:1309:server_check_marker_off] 0-glusterd: Returning 0
> [2018-10-31 17:00:51.961529] D [MSGID: 0] 
> [glusterd-volgen.c:5816:generate_brick_volfiles] 0-management: Returning 0
> [2018-10-31 17:00:51.961681] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.961756] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.961832] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.961846] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.961855] D [MSGID: 0] 
> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
> in volume moogle-gluster
> [2018-10-31 17:00:51.961864] D [MSGID: 0] 
> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.963126] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.963203] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.963280] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.963298] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.963308] D [MSGID: 0] 
> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
> in volume moogle-gluster
> [2018-10-31 17:00:51.963316] D [MSGID: 0] 
> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.964038] D [MSGID: 0] 
> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
> 0-management: Unable to find friend: sand1lian.computerisms.ca
> [2018-10-31 17:00:51.964112] D [MSGID: 0] 
> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
> [2018-10-31 17:00:51.964186] D [MSGID: 0] 
> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 is 
> local address at interface eno1
> [2018-10-31 17:00:51.964200] D [MSGID: 0] 
> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2018-10-31 17:00:51.964211] D [MSGID: 0] 
> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
> in volume moogle-gluster
> [2018-10-31 17:00:51.964226] D [MSGID: 0] 
> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.965159] D [MSGID: 0] 
> [glusterd-op-sm.c:6150:glusterd_op_commit_perform] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.965177] D [MSGID: 0] 
> [glusterd-utils.c:9664:glusterd_aggr_brick_mount_dirs] 0-management: No 
> brick_count present
> [2018-10-31 17:00:51.965193] D [MSGID: 0] 
> [glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx 
> modification not required
> [2018-10-31 17:00:51.965219] D [MSGID: 0] 
> [glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to 
> release lock of vol moogle-gluster for 
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:00:51.966350] D [MSGID: 0] 
> [glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for 
> vol moogle-gluster successfully released
> [2018-10-31 17:00:51.966462] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:00:51.966479] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:00:51.966509] D [MSGID: 0] 
> [glusterd-op-sm.c:248:glusterd_get_txn_opinfo] 0-management: 
> Successfully got opinfo for transaction ID : 
> 3f5d14c9-ee08-493c-afac-d04d53c12aad
> [2018-10-31 17:00:51.966532] D [MSGID: 0] 
> [glusterd-op-sm.c:252:glusterd_get_txn_opinfo] 0-management: Returning 0
> [2018-10-31 17:00:51.966551] D [MSGID: 0] 
> [glusterd-op-sm.c:352:glusterd_clear_txn_opinfo] 0-management: 
> Successfully cleared opinfo for transaction ID : 
> 3f5d14c9-ee08-493c-afac-d04d53c12aad
> [2018-10-31 17:00:51.966668] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> [2018-10-31 17:00:51.966561] D [MSGID: 0] 
> [glusterd-op-sm.c:356:glusterd_clear_txn_opinfo] 0-management: Returning 0
> [2018-10-31 17:00:51.966667] D [MSGID: 0] 
> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
> Returning 0
> [2018-10-31 17:00:51.968134] D [socket.c:2927:socket_event_handler] 
> 0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
> [2018-10-31 17:00:51.968183] E [MSGID: 101191] 
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
> dispatch handler
> grep: /var/lib/glusterd/vols/moogle-gluster/bricks/*: No such file or 
> directory
> [2018-10-31 17:00:51.975661] I [run.c:242:runner_log] 
>
(-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe0dbe)
> [0x7f3f248dbdbe] 
>
-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe07fe)
> [0x7f3f248db7fe] 
> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) 
> [0x7f3f28ac35a5] ) 0-management: Ran script: 
> /var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh 
> --volname=moogle-gluster
> [2018-10-31 17:01:12.466614] D [logging.c:1871:gf_log_flush_timeout_cbk] 
> 0-logging-infra: Log timer timed out. About to flush outstanding 
> messages if present
> [2018-10-31 17:01:12.466667] D 
> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
> timer now. Timeout = 120, current buf size = 5
> [2018-10-31 17:03:12.492414] D [logging.c:1871:gf_log_flush_timeout_cbk] 
> 0-logging-infra: Log timer timed out. About to flush outstanding 
> messages if present
> [2018-10-31 17:03:12.492447] D 
> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
> timer now. Timeout = 120, current buf size = 5
> 
> Not sure about the unable to find friend message:
> 
> root at sand1lian:~# dig +short sand1lian.computerisms.ca
> 192.168.25.52
> 
> start the volume:
> 
> root at sand1lian:~# gluster v start moogle-gluster
> volume start: moogle-gluster: failed: Commit failed on localhost. Please 
> check log file for details.
> 
> output of cli.log while issuing start command:
> 
> [2018-10-31 17:08:49.019079] I [cli.c:764:main] 0-cli: Started running 
> gluster with version 5.0
> [2018-10-31 17:08:49.021694] W [socket.c:3365:socket_connect] 
> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
supported"
> [2018-10-31 17:08:49.021924] W [socket.c:3365:socket_connect] 
> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
supported"
> [2018-10-31 17:08:49.101120] I [MSGID: 101190] 
> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
> with index 1
> [2018-10-31 17:08:49.101231] E [MSGID: 101191] 
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
> dispatch handler
> [2018-10-31 17:08:49.113485] I 
> [cli-rpc-ops.c:1419:gf_cli_start_volume_cbk] 0-cli: Received resp to 
> start volume
> [2018-10-31 17:08:49.113626] I [input.c:31:cli_batch] 0-: Exiting with: -1
> 
> and output of brick log while starting volume:
> 
> [2018-10-31 17:08:49.107966] I [MSGID: 100030] [glusterfsd.c:2691:main] 
> 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 5.0 
> (args: /usr/sbin/glusterfsd -s sand1lian.computerisms.ca --volfile-id 
> moogle-gluster.sand1lian.computerisms.ca.var-GlusterBrick-moogle-gluster 
> -p 
>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
> -S /var/run/gluster/f41bfcfaf40deb7d.socket --brick-name 
> /var/GlusterBrick/moogle-gluster -l 
> /var/log/glusterfs/bricks/var-GlusterBrick-moogle-gluster.log 
> --xlator-option 
> *-posix.glusterd-uuid=bb8c61eb-f321-4485-8a8d-ddc369ac2203 
> --process-name brick --brick-port 49157 --xlator-option 
> moogle-gluster-server.listen-port=49157)
> [2018-10-31 17:08:49.112123] E [socket.c:3466:socket_connect] 
> 0-glusterfs: connection attempt on? failed, (Invalid argument)
> [2018-10-31 17:08:49.112293] I [MSGID: 101190] 
> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
> with index 1
> [2018-10-31 17:08:49.112374] I [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 
> 0-glusterfsd-mgmt: disconnected from remote-host: sand1lian.computerisms.ca
> [2018-10-31 17:08:49.112399] I [glusterfsd-mgmt.c:2444:mgmt_rpc_notify] 
> 0-glusterfsd-mgmt: Exhausted all volfile servers
> [2018-10-31 17:08:49.112656] W [glusterfsd.c:1481:cleanup_and_exit] 
> (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xf023) [0x7f3466c12023] 
> -->/usr/sbin/glusterfsd(+0x1273e) [0x557f4ea6373e] 
> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x54) [0x557f4ea5be94] ) 0-: 
> received signum (1), shutting down
> [2018-10-31 17:08:49.112973] E [socket.c:3466:socket_connect] 
> 0-glusterfs: connection attempt on? failed, (Invalid argument)
> [2018-10-31 17:08:49.112996] W [rpc-clnt.c:1683:rpc_clnt_submit] 
> 0-glusterfs: error returned while attempting to connect to host:(null), 
> port:0
> [2018-10-31 17:08:49.113007] I 
> [socket.c:3710:socket_submit_outgoing_msg] 0-glusterfs: not connected 
> (priv->connected = 0)
> [2018-10-31 17:08:49.113016] W [rpc-clnt.c:1695:rpc_clnt_submit] 
> 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x2 Program: 
> Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)
> 
> 
> still seeing the empty pid file and the connection attempt on? failed, 
> (Invalid argument) as the mostly likely culprits, but have read 
> everything of relevance I have found on google and not discovered a 
> solution yet...
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On 2018-10-30 9:15 p.m., Computerisms Corporation wrote:
>> Hi,
>>
>> Fortunately I am playing in a sandbox right now, but I am good and 
>> stuck and hoping someone can point me in the right direction.
>>
>> I have been playing for about 3 months with a gluster that currently 
>> has one brick.? The idea is that I have a server with data, I need to 
>> migrate that server onto the new gluster-capable server, then I can 
>> use the original server to make a 2nd brick, then I will be able to 
>> make some room on a 3rd server for an arbiter brick.? So I am building 
>> and testing to be sure it all works before I try it in production.
>>
>> Yesterday morning I was plugging away at figuring out how to make 
>> stuff work on the new gluster server when I ran into an issue trying 
>> to rm -rf a directory and it telling me it wasn't empty when ls -al
>> showed that it was.? This has happened to me before, and what I did to 
>> fix it before was unmount the Glusterfs, go into the brick, delete the 
>> files, and remount the Glusterfs.? I did that and it appeared to mount 
>> fine, but when I tried to access the gluster mount, it gave me an 
>> error that there were too many levels of symlinks.
>>
>> I spent my day yesterday trying pretty much everything I could find on 
>> google and a few things I couldn't.? In the past when stuff has
gone
>> funny with gluster on this box, I have always shut everything down and 
>> checked if there was a new version of gluster, and indeed there was 
>> version 5.0 available.? So I did the upgrade quite early in the day. 
>> Sadly it didn't fix my problem, but it did give me an error that
led
>> me to modifying my hosts file to be ipv6 resolvable.? Also after that, 
>> the only time the gluster would mount was at reboot, but always with 
>> the symlinks error, and it wasn't really mounted as reported by
mount,
>> but the directory could be unmounted.
>>
>> Having struck out completely yesterday, today I decided to start with 
>> a new machine.? I kept a history of the commands I had used to build 
>> the gluster a few months back and pasted them all in.? Found that the 
>> 5.0 package does not enable systemd, found that I needed the ipv6 
>> entries in the hosts file again, and also found the same problem in 
>> that the glusterfs would not mount, the symlinks error at reboot, and 
>> the same log entries.
>>
>> I am still pretty new with gluster, so my best may not be that good, 
>> but as best as I can tell the issue is that the brick will not start, 
>> even with the force option.? I think the problem boils down to one or 
>> both of two lines in the logs.? In the glusterd.log I have a line:
>>
>> 0-: Unable to read pidfile: 
>>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>>
>>
>> The file exists, and I can't see anything wrong with permissions on
>> the file or the file tree leading to it, but it is a zero-bit file, so 
>> I am thinking the problem is not the file, but that it can't read
the
>> contents of the file because there aren't any.
>>
>> The other log entry is in the brick log:
>>
>> 0-glusterfs: connection attempt on? failed, (Invalid argument)
>>
>> When I looked this up, it seems in my case there should be an attempt 
>> to connect on 127.0.0.1, but given the double space I am thinking the 
>> host argument is null, hence the invalid argument.? It occurs that 
>> maybe I still need some other entry in my hosts file to satisfy this, 
>> but I can't think what it would be.? I have created DNS entries;
dig
>> works, and both hostname and FQDN resolve.
>>
>> I have tried to change a lot of things today, so probably things are 
>> buggered up beyond hope right now so even if I do find the solution 
>> maybe it won't work.? will wipe the new machine and start over
again
>> tomorrow.
>>
>> I realize the post is kinda long, sorry for that, but I want to make 
>> sure I get every thing important.? In fairness, though, I could easily 
>> double the length of this post with possibly relevant things (if you 
>> are interested).? If you are still reading, thank you so much, I would 
>> appreciate anything, even a wild guess, as to how to move forward on 
>> this?
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

My troubleshooting took me to confirming that all my package versions 
were lined up and I came to realized that I had gotten version 5.0 from 
the debian repos instead of the repo at download.gluster.org.  I 
downgraded everything to 4.1.5-1 from gluster.org, rebooted, and messed 
around a bit, and my gluster is back online.



On 2018-10-31 10:32 a.m., Computerisms Corporation
wrote:
> forgot to add output of glusterd console when starting the volume:
> 
> [2018-10-31 17:31:33.887923] D [MSGID: 0] 
> [glusterd-volume-ops.c:572:__glusterd_handle_cli_start_volume] 
> 0-management: Received start vol req for volume moogle-gluster
> [2018-10-31 17:31:33.887976] D [MSGID: 0] 
> [glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to 
> acquire lock of vol moogle-gluster for 
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:31:33.888171] D [MSGID: 0] 
> [glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for vol 
> moogle-gluster successfully held by bb8c61eb-f321-4485-8a8d-ddc369ac2203
> [2018-10-31 17:31:33.888189] D [MSGID: 0] 
> [glusterd-locks.c:519:glusterd_multiple_mgmt_v3_lock] 0-management: 
> Returning 0
> [2018-10-31 17:31:33.888204] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.888213] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888229] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.888237] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888247] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.888256] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888269] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.888277] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888294] D [MSGID: 0] 
> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
> [2018-10-31 17:31:33.888318] D [MSGID: 0] 
> [glusterd-mgmt.c:223:gd_mgmt_v3_pre_validate_fn] 0-management: OP = 5. 
> Returning 0
> [2018-10-31 17:31:33.888668] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.888682] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888719] E [MSGID: 101012] 
> [common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile: 
>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
> 
> [2018-10-31 17:31:33.888757] I 
> [glusterd-utils.c:6300:glusterd_brick_start] 0-management: starting a 
> fresh brick process for brick /var/GlusterBrick/moogle-gluster
> [2018-10-31 17:31:33.898943] D [logging.c:1998:_gf_msg_internal] 
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
> About to flush least recently used log message to disk
> [2018-10-31 17:31:33.888780] E [MSGID: 101012] 
> [common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile: 
>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
> 
> [2018-10-31 17:31:33.898942] E [MSGID: 106005] 
> [glusterd-utils.c:6305:glusterd_brick_start] 0-management: Unable to 
> start brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
> [2018-10-31 17:31:33.899068] D [MSGID: 0] 
> [glusterd-utils.c:6315:glusterd_brick_start] 0-management: returning -107
> [2018-10-31 17:31:33.899088] E [MSGID: 106122] 
> [glusterd-mgmt.c:308:gd_mgmt_v3_commit_fn] 0-management: Volume start 
> commit failed.
> [2018-10-31 17:31:33.899100] D [MSGID: 0] 
> [glusterd-mgmt.c:392:gd_mgmt_v3_commit_fn] 0-management: OP = 5. 
> Returning -107
> [2018-10-31 17:31:33.899114] E [MSGID: 106122] 
> [glusterd-mgmt.c:1557:glusterd_mgmt_v3_commit] 0-management: Commit 
> failed for operation Start on local node
> [2018-10-31 17:31:33.899128] D [MSGID: 0] 
> [glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx 
> modification not required
> [2018-10-31 17:31:33.899140] E [MSGID: 106122] 
> [glusterd-mgmt.c:2160:glusterd_mgmt_v3_initiate_all_phases] 
> 0-management: Commit Op Failed
> [2018-10-31 17:31:33.899168] D [MSGID: 0] 
> [glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to 
> release lock of vol moogle-gluster for 
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:31:33.899195] D [MSGID: 0] 
> [glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for 
> vol moogle-gluster successfully released
> [2018-10-31 17:31:33.899211] D [MSGID: 0] 
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
> moogle-gluster found
> [2018-10-31 17:31:33.899221] D [MSGID: 0] 
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.899232] D [MSGID: 0] 
> [glusterd-locks.c:464:glusterd_multiple_mgmt_v3_unlock] 0-management: 
> Returning 0
> [2018-10-31 17:31:33.899314] D [MSGID: 0] 
> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
> Returning 0
> [2018-10-31 17:31:33.900750] D [socket.c:2927:socket_event_handler] 
> 0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
> [2018-10-31 17:31:33.900809] E [MSGID: 101191] 
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
> dispatch handler
> 
> 
> On 2018-10-31 10:19 a.m., Computerisms Corporation wrote:
>> Hi,
>>
>> it occurs maybe the previous email was too many words and not enough 
>> data.? so will try to display the issue differently.
>>
>> gluster created (single brick volume following advice from 
>>
https://lists.gluster.org/pipermail/gluster-users/2016-October/028821.html):
>>
>>
>> root at sand1lian:~# gluster volume create moogle-gluster 
>> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>>
>> Gluster was started from cli with --debug, console reports the 
>> following with creation of the volume:
>>
>> [2018-10-31 17:00:51.555918] D [MSGID: 0] 
>> [glusterd-volume-ops.c:328:__glusterd_handle_create_volume] 
>> 0-management: Received create volume req
>> [2018-10-31 17:00:51.555963] D [MSGID: 0] 
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning
-1
>> [2018-10-31 17:00:51.556072] D [MSGID: 0] 
>> [glusterd-op-sm.c:209:glusterd_generate_txn_id] 0-management: 
>> Transaction_id = 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556090] D [MSGID: 0] 
>> [glusterd-op-sm.c:302:glusterd_set_txn_opinfo] 0-management: 
>> Successfully set opinfo for transaction ID : 
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556099] D [MSGID: 0] 
>> [glusterd-op-sm.c:309:glusterd_set_txn_opinfo] 0-management: Returning
0
>> [2018-10-31 17:00:51.556108] D [MSGID: 0] 
>> [glusterd-syncop.c:1809:gd_sync_task_begin] 0-management: Transaction 
>> ID : 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556127] D [MSGID: 0] 
>> [glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to 
>> acquire lock of vol moogle-gluster for 
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
>> [2018-10-31 17:00:51.556293] D [MSGID: 0] 
>> [glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for 
>> vol moogle-gluster successfully held by 
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203
>> [2018-10-31 17:00:51.556333] D [MSGID: 0] 
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning
-1
>> [2018-10-31 17:00:51.556368] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.556345] D [MSGID: 0] 
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning
-1
>> [2018-10-31 17:00:51.556368] D [MSGID: 0] 
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning
0
>> [2018-10-31 17:00:51.556608] D [MSGID: 0] 
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.556656] D [MSGID: 0] 
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556669] D [MSGID: 0] 
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556681] D [MSGID: 0] 
>> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.556690] D [MSGID: 0] 
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556699] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy]
0-:
>> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556690]
and
>> [2018-10-31 17:00:51.556698]
>> [2018-10-31 17:00:51.556699] D [MSGID: 0] 
>> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning
0
>> [2018-10-31 17:00:51.556728] D [MSGID: 0] 
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning
0
>> [2018-10-31 17:00:51.556738] D [MSGID: 0] 
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.556752] D [MSGID: 0] 
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556764] D [MSGID: 0] 
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556772] D [MSGID: 0] 
>> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.556781] D [MSGID: 0] 
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556791] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy]
0-:
>> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556781]
and
>> [2018-10-31 17:00:51.556790]
>> [2018-10-31 17:00:51.556791] D [MSGID: 0] 
>> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning
0
>> [2018-10-31 17:00:51.556818] D [MSGID: 0] 
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning
0
>> [2018-10-31 17:00:51.556955] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557033] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557140] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557154] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.557172] D [MSGID: 0] 
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557183] D [MSGID: 0] 
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning
0
>> [2018-10-31 17:00:51.557198] D [MSGID: 0] 
>> [glusterd-utils.c:7558:glusterd_new_brick_validate] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.557207] D [MSGID: 0] 
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning
0
>> [2018-10-31 17:00:51.557392] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557468] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557542] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557554] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.557573] D [MSGID: 0] 
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.557586] D [MSGID: 0] 
>> [glusterd-volume-ops.c:1467:glusterd_op_stage_create_volume] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557595] D [MSGID: 0] 
>> [glusterd-op-sm.c:6014:glusterd_op_stage_validate] 0-management: OP = 
>> 1. Returning 0
>> [2018-10-31 17:00:51.557610] D [MSGID: 0] 
>> [glusterd-op-sm.c:7659:glusterd_op_bricks_select] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.557620] D [MSGID: 0] 
>> [glusterd-syncop.c:1751:gd_brick_op_phase] 0-management: Sent op req 
>> to 0 bricks
>> [2018-10-31 17:00:51.557663] D [MSGID: 0] 
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.557693] D [MSGID: 0] 
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning
0
>> [2018-10-31 17:00:51.557771] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557844] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557917] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557931] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.557947] D [MSGID: 0] 
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557957] D [MSGID: 0] 
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning
0
>> [2018-10-31 17:00:51.558393] D [MSGID: 0] 
>> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning 0
>> [2018-10-31 17:00:51.558409] D [MSGID: 0] 
>> [glusterd-volgen.c:3140:_get_xlator_opt_key_from_vme] 0-glusterd: 
>> Returning 0
>> [2018-10-31 17:00:51.558495] W [MSGID: 101095] 
>> [xlator.c:180:xlator_volopt_dynload] 0-xlator: 
>> /usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/nfs/server.so: cannot 
>> open shared object file: No such file or directory
>> [2018-10-31 17:00:51.558509] D [MSGID: 0] 
>> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning -1
>> [2018-10-31 17:00:51.558566] D [MSGID: 0] 
>> [glusterd-store.c:1107:glusterd_store_create_volume_dir] 0-management: 
>> Returning with 0
>> [2018-10-31 17:00:51.558593] D [MSGID: 0] 
>> [glusterd-store.c:1125:glusterd_store_create_volume_run_dir] 
>> 0-management: Returning with 0
>> [2018-10-31 17:00:51.899586] D [MSGID: 0] 
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.930562] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.930485] D [MSGID: 0] 
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.930561] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.932563] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
>> 0-management: returning: 0" repeated 19 times between [2018-10-31 
>> 17:00:51.930561] and [2018-10-31 17:00:51.930794]
>> [2018-10-31 17:00:51.932562] D [MSGID: 0] 
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.932688] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.932709] D [MSGID: 0] 
>> [glusterd-store.c:457:glusterd_store_snapd_write] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.935196] D [MSGID: 0] 
>> [glusterd-store.c:521:glusterd_store_perform_snapd_store] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.935226] D [MSGID: 0] 
>> [glusterd-store.c:585:glusterd_store_snapd_info] 0-management: 
>> Returning with 0
>> [2018-10-31 17:00:51.935251] D [MSGID: 0] 
>> [glusterd-store.c:788:_storeopts] 0-management: Storing in 
>> volinfo:key= transport.address-family, val=inet
>> [2018-10-31 17:00:51.935290] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.935314] D [MSGID: 0] 
>> [glusterd-store.c:788:_storeopts] 0-management: Storing in 
>> volinfo:key= nfs.disable, val=on
>> [2018-10-31 17:00:51.935344] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.935360] D [MSGID: 0] 
>> [glusterd-store.c:1174:glusterd_store_volinfo_write] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.935382] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.936584] D [MSGID: 0] 
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.936685] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.936807] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
>> 0-management: returning: 0" repeated 10 times between [2018-10-31 
>> 17:00:51.936685] and [2018-10-31 17:00:51.936806]
>> [2018-10-31 17:00:51.936807] D [MSGID: 0] 
>> [glusterd-store.c:430:glusterd_store_brickinfo_write] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.936833] D [MSGID: 0] 
>> [glusterd-store.c:481:glusterd_store_perform_brick_store] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.936841] D [MSGID: 0] 
>> [glusterd-store.c:550:glusterd_store_brickinfo] 0-management: 
>> Returning with 0
>> [2018-10-31 17:00:51.936848] D [MSGID: 0] 
>> [glusterd-store.c:1394:glusterd_store_brickinfos] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.936856] D [MSGID: 0] 
>> [glusterd-store.c:1620:glusterd_store_perform_volume_store] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.958353] D [MSGID: 0] 
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.958494] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value] 
>> 0-management: returning: 0" repeated 9 times between [2018-10-31 
>> 17:00:51.958353] and [2018-10-31 17:00:51.958493]
>> [2018-10-31 17:00:51.958493] D [MSGID: 0] 
>> [glusterd-store.c:1558:glusterd_store_node_state_write] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.960449] D [MSGID: 0] 
>> [glusterd-store.c:1592:glusterd_store_perform_node_state_store] 
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.960683] D [MSGID: 0] 
>> [glusterd-utils.c:2840:glusterd_volume_compute_cksum] 0-management: 
>> Returning with 0
>> [2018-10-31 17:00:51.960699] D [MSGID: 0] 
>> [glusterd-store.c:1832:glusterd_store_volinfo] 0-management: Returning
0
>> [2018-10-31 17:00:51.960797] D [MSGID: 0] 
>> [glusterd-utils.c:181:_brick_for_each] 0-management: Found a brick - 
>> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>> [2018-10-31 17:00:51.961200] D [MSGID: 0] 
>> [glusterd-volgen.c:1309:server_check_marker_off] 0-glusterd: Returning
0
>> [2018-10-31 17:00:51.961529] D [MSGID: 0] 
>> [glusterd-volgen.c:5816:generate_brick_volfiles] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.961681] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.961756] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.961832] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.961846] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.961855] D [MSGID: 0] 
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.961864] D [MSGID: 0] 
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.963126] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.963203] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.963280] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.963298] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.963308] D [MSGID: 0] 
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.963316] D [MSGID: 0] 
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.964038] D [MSGID: 0] 
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname] 
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.964112] D [MSGID: 0] 
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.964186] D [MSGID: 0] 
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52 
>> is local address at interface eno1
>> [2018-10-31 17:00:51.964200] D [MSGID: 0] 
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management: 
>> returning 0
>> [2018-10-31 17:00:51.964211] D [MSGID: 0] 
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management: 
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster 
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.964226] D [MSGID: 0] 
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.965159] D [MSGID: 0] 
>> [glusterd-op-sm.c:6150:glusterd_op_commit_perform] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.965177] D [MSGID: 0] 
>> [glusterd-utils.c:9664:glusterd_aggr_brick_mount_dirs] 0-management: 
>> No brick_count present
>> [2018-10-31 17:00:51.965193] D [MSGID: 0] 
>> [glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx 
>> modification not required
>> [2018-10-31 17:00:51.965219] D [MSGID: 0] 
>> [glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to 
>> release lock of vol moogle-gluster for 
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
>> [2018-10-31 17:00:51.966350] D [MSGID: 0] 
>> [glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for 
>> vol moogle-gluster successfully released
>> [2018-10-31 17:00:51.966462] D [MSGID: 0] 
>> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume 
>> moogle-gluster found
>> [2018-10-31 17:00:51.966479] D [MSGID: 0] 
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
>> [2018-10-31 17:00:51.966509] D [MSGID: 0] 
>> [glusterd-op-sm.c:248:glusterd_get_txn_opinfo] 0-management: 
>> Successfully got opinfo for transaction ID : 
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.966532] D [MSGID: 0] 
>> [glusterd-op-sm.c:252:glusterd_get_txn_opinfo] 0-management: Returning
0
>> [2018-10-31 17:00:51.966551] D [MSGID: 0] 
>> [glusterd-op-sm.c:352:glusterd_clear_txn_opinfo] 0-management: 
>> Successfully cleared opinfo for transaction ID : 
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.966668] D [logging.c:1998:_gf_msg_internal] 
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.966561] D [MSGID: 0] 
>> [glusterd-op-sm.c:356:glusterd_clear_txn_opinfo] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.966667] D [MSGID: 0] 
>> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management: 
>> Returning 0
>> [2018-10-31 17:00:51.968134] D [socket.c:2927:socket_event_handler] 
>> 0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
>> [2018-10-31 17:00:51.968183] E [MSGID: 101191] 
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
>> dispatch handler
>> grep: /var/lib/glusterd/vols/moogle-gluster/bricks/*: No such file or 
>> directory
>> [2018-10-31 17:00:51.975661] I [run.c:242:runner_log] 
>>
(-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe0dbe)
>> [0x7f3f248dbdbe] 
>>
-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe07fe)
>> [0x7f3f248db7fe] 
>> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) 
>> [0x7f3f28ac35a5] ) 0-management: Ran script: 
>> /var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh 
>> --volname=moogle-gluster
>> [2018-10-31 17:01:12.466614] D 
>> [logging.c:1871:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer 
>> timed out. About to flush outstanding messages if present
>> [2018-10-31 17:01:12.466667] D 
>> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
>> timer now. Timeout = 120, current buf size = 5
>> [2018-10-31 17:03:12.492414] D 
>> [logging.c:1871:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer 
>> timed out. About to flush outstanding messages if present
>> [2018-10-31 17:03:12.492447] D 
>> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting 
>> timer now. Timeout = 120, current buf size = 5
>>
>> Not sure about the unable to find friend message:
>>
>> root at sand1lian:~# dig +short sand1lian.computerisms.ca
>> 192.168.25.52
>>
>> start the volume:
>>
>> root at sand1lian:~# gluster v start moogle-gluster
>> volume start: moogle-gluster: failed: Commit failed on localhost. 
>> Please check log file for details.
>>
>> output of cli.log while issuing start command:
>>
>> [2018-10-31 17:08:49.019079] I [cli.c:764:main] 0-cli: Started running 
>> gluster with version 5.0
>> [2018-10-31 17:08:49.021694] W [socket.c:3365:socket_connect] 
>> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not 
>> supported"
>> [2018-10-31 17:08:49.021924] W [socket.c:3365:socket_connect] 
>> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not 
>> supported"
>> [2018-10-31 17:08:49.101120] I [MSGID: 101190] 
>> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started 
>> thread with index 1
>> [2018-10-31 17:08:49.101231] E [MSGID: 101191] 
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
>> dispatch handler
>> [2018-10-31 17:08:49.113485] I 
>> [cli-rpc-ops.c:1419:gf_cli_start_volume_cbk] 0-cli: Received resp to 
>> start volume
>> [2018-10-31 17:08:49.113626] I [input.c:31:cli_batch] 0-: Exiting 
>> with: -1
>>
>> and output of brick log while starting volume:
>>
>> [2018-10-31 17:08:49.107966] I [MSGID: 100030] 
>> [glusterfsd.c:2691:main] 0-/usr/sbin/glusterfsd: Started running 
>> /usr/sbin/glusterfsd version 5.0 (args: /usr/sbin/glusterfsd -s 
>> sand1lian.computerisms.ca --volfile-id 
>>
moogle-gluster.sand1lian.computerisms.ca.var-GlusterBrick-moogle-gluster
>> -p 
>>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>> -S /var/run/gluster/f41bfcfaf40deb7d.socket --brick-name 
>> /var/GlusterBrick/moogle-gluster -l 
>> /var/log/glusterfs/bricks/var-GlusterBrick-moogle-gluster.log 
>> --xlator-option 
>> *-posix.glusterd-uuid=bb8c61eb-f321-4485-8a8d-ddc369ac2203 
>> --process-name brick --brick-port 49157 --xlator-option 
>> moogle-gluster-server.listen-port=49157)
>> [2018-10-31 17:08:49.112123] E [socket.c:3466:socket_connect] 
>> 0-glusterfs: connection attempt on? failed, (Invalid argument)
>> [2018-10-31 17:08:49.112293] I [MSGID: 101190] 
>> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started 
>> thread with index 1
>> [2018-10-31 17:08:49.112374] I 
>> [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 0-glusterfsd-mgmt: 
>> disconnected from remote-host: sand1lian.computerisms.ca
>> [2018-10-31 17:08:49.112399] I 
>> [glusterfsd-mgmt.c:2444:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted 
>> all volfile servers
>> [2018-10-31 17:08:49.112656] W [glusterfsd.c:1481:cleanup_and_exit] 
>> (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xf023)
[0x7f3466c12023]
>> -->/usr/sbin/glusterfsd(+0x1273e) [0x557f4ea6373e] 
>> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x54) [0x557f4ea5be94] )
0-:
>> received signum (1), shutting down
>> [2018-10-31 17:08:49.112973] E [socket.c:3466:socket_connect] 
>> 0-glusterfs: connection attempt on? failed, (Invalid argument)
>> [2018-10-31 17:08:49.112996] W [rpc-clnt.c:1683:rpc_clnt_submit] 
>> 0-glusterfs: error returned while attempting to connect to 
>> host:(null), port:0
>> [2018-10-31 17:08:49.113007] I 
>> [socket.c:3710:socket_submit_outgoing_msg] 0-glusterfs: not connected 
>> (priv->connected = 0)
>> [2018-10-31 17:08:49.113016] W [rpc-clnt.c:1695:rpc_clnt_submit] 
>> 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x2 
>> Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport 
>> (glusterfs)
>>
>>
>> still seeing the empty pid file and the connection attempt on? failed, 
>> (Invalid argument) as the mostly likely culprits, but have read 
>> everything of relevance I have found on google and not discovered a 
>> solution yet...
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 2018-10-30 9:15 p.m., Computerisms Corporation wrote:
>>> Hi,
>>>
>>> Fortunately I am playing in a sandbox right now, but I am good and 
>>> stuck and hoping someone can point me in the right direction.
>>>
>>> I have been playing for about 3 months with a gluster that
currently
>>> has one brick.? The idea is that I have a server with data, I need
to
>>> migrate that server onto the new gluster-capable server, then I can
>>> use the original server to make a 2nd brick, then I will be able to
>>> make some room on a 3rd server for an arbiter brick.? So I am 
>>> building and testing to be sure it all works before I try it in 
>>> production.
>>>
>>> Yesterday morning I was plugging away at figuring out how to make 
>>> stuff work on the new gluster server when I ran into an issue
trying
>>> to rm -rf a directory and it telling me it wasn't empty when ls
-al
>>> showed that it was.? This has happened to me before, and what I did
>>> to fix it before was unmount the Glusterfs, go into the brick,
delete
>>> the files, and remount the Glusterfs.? I did that and it appeared
to
>>> mount fine, but when I tried to access the gluster mount, it gave
me
>>> an error that there were too many levels of symlinks.
>>>
>>> I spent my day yesterday trying pretty much everything I could find
>>> on google and a few things I couldn't.? In the past when stuff
has
>>> gone funny with gluster on this box, I have always shut everything 
>>> down and checked if there was a new version of gluster, and indeed 
>>> there was version 5.0 available.? So I did the upgrade quite early
in
>>> the day. Sadly it didn't fix my problem, but it did give me an
error
>>> that led me to modifying my hosts file to be ipv6 resolvable.? Also
>>> after that, the only time the gluster would mount was at reboot,
but
>>> always with the symlinks error, and it wasn't really mounted as
>>> reported by mount, but the directory could be unmounted.
>>>
>>> Having struck out completely yesterday, today I decided to start
with
>>> a new machine.? I kept a history of the commands I had used to
build
>>> the gluster a few months back and pasted them all in.? Found that
the
>>> 5.0 package does not enable systemd, found that I needed the ipv6 
>>> entries in the hosts file again, and also found the same problem in
>>> that the glusterfs would not mount, the symlinks error at reboot,
and
>>> the same log entries.
>>>
>>> I am still pretty new with gluster, so my best may not be that
good,
>>> but as best as I can tell the issue is that the brick will not
start,
>>> even with the force option.? I think the problem boils down to one
or
>>> both of two lines in the logs.? In the glusterd.log I have a line:
>>>
>>> 0-: Unable to read pidfile: 
>>>
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>>>
>>>
>>> The file exists, and I can't see anything wrong with
permissions on
>>> the file or the file tree leading to it, but it is a zero-bit file,
>>> so I am thinking the problem is not the file, but that it can't
read
>>> the contents of the file because there aren't any.
>>>
>>> The other log entry is in the brick log:
>>>
>>> 0-glusterfs: connection attempt on? failed, (Invalid argument)
>>>
>>> When I looked this up, it seems in my case there should be an
attempt
>>> to connect on 127.0.0.1, but given the double space I am thinking
the
>>> host argument is null, hence the invalid argument.? It occurs that 
>>> maybe I still need some other entry in my hosts file to satisfy
this,
>>> but I can't think what it would be.? I have created DNS
entries; dig
>>> works, and both hostname and FQDN resolve.
>>>
>>> I have tried to change a lot of things today, so probably things
are
>>> buggered up beyond hope right now so even if I do find the solution
>>> maybe it won't work.? will wipe the new machine and start over
again
>>> tomorrow.
>>>
>>> I realize the post is kinda long, sorry for that, but I want to
make
>>> sure I get every thing important.? In fairness, though, I could 
>>> easily double the length of this post with possibly relevant things
>>> (if you are interested).? If you are still reading, thank you so 
>>> much, I would appreciate anything, even a wild guess, as to how to 
>>> move forward on this?
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users