陈曦
2018-Jan-14 07:43 UTC
[Gluster-users] Volume can not write to data if this volume quota limits capacity and mount itself volume on arm64(aarch64) architecture
Thanks for reading this email?I found a problem while using Glusterfs? First?I created a Distributed Dispersed volume on three nodes?and Limit the volume capacity use quota command?this volume is auto mounted on /run/gluster/VOLUME_NAME. This volume can be read and written normally? After, I manually mounted the volume in another path to provide data storage of SAMBA and ISCSI services, after which the volume can not be read and written normal? ?What is the auto mount parameter ?the /run/gluster/VOLUME_NAME?? can I write it to /etc/ fstab and use this mount point directly? this is my commond and environment [root at f08n29glusterfs]# uname -a Linux f08n29 4.2.0 #1 SMP Tue Jun 7 01:18:20 CST 2016 aarch64 aarch64 aarch64 GNU/Linux [root at f08n29glusterfs]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (AltArch) [root at f08n25test_vol]# glusterfs --version glusterfs 3.7.20 root at f08n25test_vol]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.33.0.7 f08n25 10.33.0.8 f08n33 10.33.0.30 f08n29 [root at f08n29glusterfs]# gluster pool list UUID Hostname State 05b203e0-6a2d-44bf-bce0-c1a5dbf2e81f 10.33.0.8 Connected c72285e0-b98d-4a50-afa7-677916d1b2bf 10.33.0.7 Connected a3de3c39-7574-4f6c-9429-de8ee24a97ee localhost Connected [root at f08n29glusterfs]# gluster volume create test_vol disperse 3 redundancy 1 BRICK_LIST_ON_THREE_NODE force [root at f08n29glusterfs]# gluster volume start test_vol [root at f08n29glusterfs]# gluster volume quota test_vol enable [root at f08n29glusterfs]# gluster volume quota test_vol limit-usage / 51000.0GB [root at f08n29glusterfs]# mount.glusterfs 10.33.0.7:/test_vol /data/gluster/test_vol [root at f08n29glusterfs]# cd /data/gluster/test_vol [root at f08n25glusterfs]# mount ...... localhost:test_vol on /run/gluster/test_vol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) localhost:/test_vol on /data/gluster/test_vol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root at f08n25glusterfs]# gluster volume status Status of volume: test_vol Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.33.0.30:/data/gluster/11ddcbb9-069 c-4de8-b3c1-cb11c9c1f770/test_vol 49210 0 Y 17870 ...... NFS Server on localhost 2049 0 Y 78928 Self-heal Daemon on localhost N/A N/A Y 78935 Quota Daemon on localhost N/A N/A Y 78972 NFS Server on f08n29 2049 0 Y 18100 Self-heal Daemon on f08n29 N/A N/A Y 18107 Quota Daemon on f08n29 N/A N/A Y 18211 NFS Server on 10.33.0.8 2049 0 Y 90323 Self-heal Daemon on 10.33.0.8 N/A N/A Y 90330 Quota Daemon on 10.33.0.8 N/A N/A Y 90383 Task Status of Volume test_vol ------------------------------------------------------------------------------ There are no active volume tasks [root at f08n29mnt]# mkdir 111 mkdir: cannot create directory ?111?: Invalid argument [root at f08n25test_vol]# ls ls: cannot access 111: No such file or directory 111 [root at f08n25test_vol]# touch file touch: setting times of ?file?: No such file or directory Using other nodes also can not write data normally this is some log [root at f08n25glusterfs]# tail -n 100 test_vol-quota-crawl.log [2018-02-02 11:23:31.419152] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.419274] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.419330] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-0: Heal failed [Invalid argument] [2018-02-02 11:23:31.419703] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.419744] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.419772] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument] [2018-02-02 11:23:31.420085] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.420141] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.420184] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-2: Heal failed [Invalid argument] [2018-02-02 11:23:31.420401] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.420516] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.420551] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-3: Heal failed [Invalid argument] [2018-02-02 11:23:31.420924] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.420965] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.420996] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-4: Heal failed [Invalid argument] [2018-02-02 11:23:31.421487] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.421527] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.421554] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-6: Heal failed [Invalid argument] [2018-02-02 11:23:31.421596] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.421696] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.421730] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-7: Heal failed [Invalid argument] [2018-02-02 11:23:31.421901] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.421957] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.421991] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-8: Heal failed [Invalid argument] [2018-02-02 11:23:31.422084] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.422138] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.422171] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument] [2018-02-02 11:23:31.422224] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.422263] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.422291] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:23:31.422360] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.422400] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.422427] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-10: Heal failed [Invalid argument] [2018-02-02 11:23:31.422588] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.422629] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.422661] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-11: Heal failed [Invalid argument] [2018-02-02 11:23:31.422726] I [MSGID: 109063] [dht-layout.c:718:dht_layout_normalize] 0-test_vol-dht: Found anomalies in /.trashcan (gfid = 00000000-0000-0000-0000-000000000005). Holes=1 overlaps=0 [2018-02-02 11:23:31.435685] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.435809] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.436116] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.436156] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.436461] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.436514] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.436898] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.436995] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.437325] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.437373] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.437779] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.437820] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.438006] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.438059] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.438152] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.438241] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.438451] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.438497] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.438603] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.438746] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.438752] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.438880] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.439070] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.439126] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.439301] I [MSGID: 109036] [dht-common.c:9082:dht_log_new_layout_for_dir_selfheal] 0-test_vol-dht: Setting layout of /.trashcan with [Subvol_name: test_vol-disperse-0, Err: -1 , Start: 2863311528 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: test_vol-disperse-1, Err: -1 , Start: 3221225469 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: test_vol-disperse-10, Err: -1 , Start: 3579139410 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: test_vol-disperse-11, Err: -1 , Start: 3937053351 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: test_vol-disperse-2, Err: -1 , Start: 0 , Stop: 357913940 , Hash: 1 ], [Subvol_name: test_vol-disperse-3, Err: -1 , Start: 357913941 , Stop: 715827881 , Hash: 1 ], [Subvol_name: test_vol-disperse-4, Err: -1 , Start: 715827882 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: test_vol-disperse-5, Err: -1 , Start: 1073741823 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: test_vol-disperse-6, Err: -1 , Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: test_vol-disperse-7, Err: -1 , Start: 1789569705 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: test_vol-disperse-8, Err: -1 , Start: 2147483646 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: test_vol-disperse-9, Err: -1 , Start: 2505397587 , Stop: 2863311527 , Hash: 1 ], [2018-02-02 11:23:31.441413] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-1: /.trashcan: name heal successful on 7 [2018-02-02 11:23:31.442588] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-4: /.trashcan: name heal successful on 7 [2018-02-02 11:23:31.445959] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-3: /.trashcan: name heal successful on 7 [2018-02-02 11:23:31.476532] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.476584] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.476615] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument] [2018-02-02 11:23:31.478545] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.478606] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.478644] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-10: Heal failed [Invalid argument] [2018-02-02 11:23:31.478823] I [MSGID: 109063] [dht-layout.c:718:dht_layout_normalize] 0-test_vol-dht: Found anomalies in /.trashcan/internal_op (gfid = 00000000-0000-0000-0000-000000000006). Holes=1 overlaps=0 [2018-02-02 11:23:31.491605] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.491665] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.491967] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.492007] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.492301] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.492341] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.492723] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.492768] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.493121] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.493162] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.493558] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.493599] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.494023] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.494067] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.494107] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.494154] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.494393] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.494493] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.494798] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.494952] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.494991] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.494998] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.495365] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP' [2018-02-02 11:23:31.495406] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:23:31.495542] I [MSGID: 109036] [dht-common.c:9082:dht_log_new_layout_for_dir_selfheal] 0-test_vol-dht: Setting layout of /.trashcan/internal_op with [Subvol_name: test_vol-disperse-0, Err: -1 , Start: 715827882 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: test_vol-disperse-1, Err: -1 , Start: 1073741823 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: test_vol-disperse-10, Err: -1 , Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: test_vol-disperse-11, Err: -1 , Start: 1789569705 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: test_vol-disperse-2, Err: -1 , Start: 2147483646 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: test_vol-disperse-3, Err: -1 , Start: 2505397587 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: test_vol-disperse-4, Err: -1 , Start: 2863311528 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: test_vol-disperse-5, Err: -1 , Start: 3221225469 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: test_vol-disperse-6, Err: -1 , Start: 3579139410 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: test_vol-disperse-7, Err: -1 , Start: 3937053351 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: test_vol-disperse-8, Err: -1 , Start: 0 , Stop: 357913940 , Hash: 1 ], [Subvol_name: test_vol-disperse-9, Err: -1 , Start: 357913941 , Stop: 715827881 , Hash: 1 ], [2018-02-02 11:23:31.527002] I [fuse-bridge.c:5016:fuse_thread_proc] 0-fuse: unmounting /tmp/mntEHHMn8 [2018-02-02 11:23:31.527167] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7bb0) [0xffff820a7bb0] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x11c) [0xaaaacbcc9bc0] -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaacbcc99fc] ) 0-: received signum (15), shutting down [2018-02-02 11:23:31.527206] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/tmp/mntEHHMn8'. [root at f08n25glusterfs]# tail -n 100 data-gluster-test_vol.log 719: end-volume 720: 721: volume meta-autoload 722: type meta 723: subvolumes test_vol 724: end-volume 725: +------------------------------------------------------------------------------+ [2018-02-02 11:23:36.282441] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-33: Connected to test_vol-client-33, attached to remote volume '/data/gluster/d96a32f0-cfb9-4ee4-b3a2-54ae9091fcb0/test_vol'. [2018-02-02 11:23:36.282517] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-33: Server and Client lk-version numbers are not same, reopening the fds [2018-02-02 11:23:36.282887] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-33: Server lk version = 1 [2018-02-02 11:23:36.283198] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-test_vol-client-35: changing port to 49221 (from 0) [2018-02-02 11:23:36.283668] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-test_vol-client-34: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-02 11:23:36.284446] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-34: Connected to test_vol-client-34, attached to remote volume '/data/gluster/eca8cd11-d264-40ab-a384-4f8f56fc1e4f/test_vol'. [2018-02-02 11:23:36.284501] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-34: Server and Client lk-version numbers are not same, reopening the fds [2018-02-02 11:23:36.284742] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-34: Server lk version = 1 [2018-02-02 11:23:36.291994] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-test_vol-client-35: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-02 11:23:36.292683] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-35: Connected to test_vol-client-35, attached to remote volume '/data/gluster/f6c653ed-1665-43cd-8a0c-8eaa8e23506b/test_vol'. [2018-02-02 11:23:36.292720] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-35: Server and Client lk-version numbers are not same, reopening the fds [2018-02-02 11:23:36.292830] I [MSGID: 122061] [ec.c:313:ec_up] 0-test_vol-disperse-11: Going UP [2018-02-02 11:23:36.297432] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-35: Server lk version = 1 [2018-02-02 11:23:36.297606] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.23 [2018-02-02 11:23:36.297662] I [fuse-bridge.c:4768:fuse_graph_sync] 0-fuse: switched to graph 0 [2018-02-02 11:23:39.223315] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-02-02 11:23:39.449492] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-02-02 11:23:39.516999] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2018-02-02 11:23:39.542305] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2018-02-02 11:24:46.657381] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:46.657536] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:46.657598] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:46.657630] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:46.657892] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 17: MKDIR() /111 => -1 (Invalid argument) [2018-02-02 11:24:55.295010] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.295077] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.298071] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.298165] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.299100] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.299151] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.302045] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.302083] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.303116] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.303153] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.305829] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.305879] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.309790] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:55.309980] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:55.310040] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:55.310070] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:55.310331] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 20: MKDIR() /111 => -1 (Invalid argument) [2018-02-02 11:24:59.853932] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.853993] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.856669] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.856709] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.857768] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.857806] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.860562] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.860599] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.861577] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.861613] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.864300] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.864345] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.867964] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-16: remote operation failed. Path: /111 [Is a directory] [2018-02-02 11:24:59.867992] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:59.868134] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 11:24:59.868191] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:24:59.868221] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:24:59.868301] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 23: /111 => -1 (Invalid argument) [2018-02-02 11:25:03.461970] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:03.462035] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:03.464735] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:03.464784] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:03.465810] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:03.465846] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:03.468554] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:03.468590] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:05.167076] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:05.167130] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:05.169788] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:05.169826] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:05.170867] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:05.170904] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 11:25:05.173580] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 11:25:05.173629] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 12:24:01.295847] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 12:24:01.295914] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument] [2018-02-02 12:24:19.680348] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 12:24:19.680410] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument] [2018-02-02 12:24:19.694268] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 12:24:19.694280] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-16: remote operation failed. Path: /111 [Is a directory] [2018-02-02 12:24:19.694438] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument] [2018-02-02 12:24:19.694555] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 187: /111 => -1 (Invalid argument) The message "W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]" repeated 6 times between [2018-02-02 12:24:19.680410] and [2018-02-02 12:24:19.694503] [2018-02-02 12:24:31.428284] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-3: remote operation failed. Path: /file [Invalid argument] [2018-02-02 12:24:31.428435] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-5: remote operation failed. Path: /file [Invalid argument] [2018-02-02 12:24:31.428505] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2) [2018-02-02 12:24:31.428541] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument] [2018-02-02 12:24:31.428624] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 192: /file => -1 (Invalid argument) The message "W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)" repeated 3 times between [2018-02-02 12:24:01.295847] and [2018-02-02 12:24:01.302362] The message "W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument]" repeated 3 times between [2018-02-02 12:24:01.295914] and [2018-02-02 12:24:01.302365] The message "W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)" repeated 6 times between [2018-02-02 12:24:19.680348] and [2018-02-02 12:24:19.694499] [root at f08n25bricks]# tail -n 100 data-gluster-1045fe63-bf09-42f5-986b-ce2d3d63def0-test_vol.log [2018-02-02 11:23:28.615522] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:28.615573] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90322-2018/01/14-06:17:22:557038-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:28.730628] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:28.730683] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18106-2018/01/14-06:17:25:563091-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:28.820630] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:28.820686] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18099-2018/01/14-06:17:25:551541-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:29.028758] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.028816] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-78971-2018/02/02-11:23:28:906426-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:29.492501] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.492553] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-24929-2018/01/13-10:10:52:662544-test_vol-client-1-19-0 (version: 3.7.20) [2018-02-02 11:23:29.805720] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-02-02 11:23:29.815869] I [graph.c:269:gf_add_cmdline_options] 0-test_vol-server: adding option 'listen-port' for volume 'test_vol-server' with value '49163' [2018-02-02 11:23:29.815936] I [graph.c:269:gf_add_cmdline_options] 0-test_vol-posix: adding option 'glusterd-uuid' for volume 'test_vol-posix' with value 'c72285e0-b98d-4a50-afa7-677916d1b2bf' [2018-02-02 11:23:29.816237] I [MSGID: 121037] [changetimerecorder.c:1967:reconfigure] 0-test_vol-changetimerecorder: set [2018-02-02 11:23:29.816393] I [MSGID: 0] [gfdb_sqlite3.c:1374:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 1000 [2018-02-02 11:23:29.816465] I [MSGID: 0] [gfdb_sqlite3.c:1374:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 1000 [2018-02-02 11:23:29.816947] W [socket.c:3677:reconfigure] 0-test_vol-quota: NBIO on -1 failed (Bad file descriptor) [2018-02-02 11:23:29.817163] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol: skip format check for non-addr auth option auth.login./data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol.allow [2018-02-02 11:23:29.817167] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol: skip format check for non-addr auth option auth.login.ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16.password [2018-02-02 11:23:29.817424] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817472] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817515] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817554] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817594] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817633] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817675] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817717] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817763] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817808] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817896] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817941] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.817985] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.818026] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.864818] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.864868] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13570-2018/01/13-10:33:03:552250-test_vol-client-1-13-1 (version: 3.7.20) [2018-02-02 11:23:29.869630] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.869665] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7618-2018/01/13-10:11:59:173084-test_vol-client-1-15-1 (version: 3.7.20) [2018-02-02 11:23:29.942042] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:29.942100] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:30.627852] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:30.627912] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18210-2018/01/14-06:17:27:450728-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:30.645858] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:30.645908] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90382-2018/01/14-06:17:24:698757-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:30.895296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:30.895347] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13495-2018/01/13-10:33:01:87712-test_vol-client-1-15-0 (version: 3.7.20) [2018-02-02 11:23:31.260628] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.260673] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-25170-2018/01/13-10:12:01:772309-test_vol-client-1-17-0 (version: 3.7.20) [2018-02-02 11:23:31.531469] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0 [2018-02-02 11:23:31.531543] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000001 held by {client=0xffff9c105920, pid=-100 lk-owner=6c005a69ffff0000} [2018-02-02 11:23:31.531577] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c105920, pid=-100 lk-owner=68055a69ffff0000} [2018-02-02 11:23:31.531604] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c105920, pid=-100 lk-owner=640a5a69ffff0000} [2018-02-02 11:23:31.531651] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on / [2018-02-02 11:23:31.531717] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan [2018-02-02 11:23:31.531767] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op [2018-02-02 11:23:31.531829] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0 [2018-02-02 11:23:31.650679] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.650736] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-24929-2018/01/13-10:10:52:662544-test_vol-client-1-23-0 (version: 3.7.20) [2018-02-02 11:23:31.680844] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.680891] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:31.696391] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.696440] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7618-2018/01/13-10:11:59:173084-test_vol-client-1-21-0 (version: 3.7.20) [2018-02-02 11:23:31.715252] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.715301] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7595-2018/01/13-10:10:50:88945-test_vol-client-1-23-0 (version: 3.7.20) [2018-02-02 11:23:31.746750] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.746797] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:31.826695] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:31.826752] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13570-2018/01/13-10:33:03:552250-test_vol-client-1-19-0 (version: 3.7.20) [2018-02-02 11:23:32.101199] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:32.101245] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13495-2018/01/13-10:33:01:87712-test_vol-client-1-19-0 (version: 3.7.20) [2018-02-02 11:23:32.178380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:32.178428] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-25170-2018/01/13-10:12:01:772309-test_vol-client-1-21-0 (version: 3.7.20) [2018-02-02 11:23:32.863582] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:32.863672] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-79117-2018/02/02-11:23:32:733518-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:35.796028] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:35.796087] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18341-2018/01/14-06:17:32:639576-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:35.802029] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:35.802064] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-79181-2018/02/02-11:23:35:671964-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:35.829726] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16 [2018-02-02 11:23:35.829775] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90605-2018/01/14-06:17:29:876725-test_vol-client-1-0-0 (version: 3.7.20) [2018-02-02 11:23:36.842225] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0 [2018-02-02 11:23:36.842297] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c109f50, pid=-100 lk-owner=6c00af7effff0000} [2018-02-02 11:23:36.842330] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c109f50, pid=-100 lk-owner=6805af7effff0000} [2018-02-02 11:23:36.842366] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on / [2018-02-02 11:23:36.842422] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan [2018-02-02 11:23:36.842524] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op [2018-02-02 11:23:36.842621] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0 [2018-02-02 11:23:36.946546] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0 [2018-02-02 11:23:36.946631] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000001 held by {client=0xffff9c1123e0, pid=-100 lk-owner=6c006587ffff0000} [2018-02-02 11:23:36.946664] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c1123e0, pid=-100 lk-owner=68056587ffff0000} [2018-02-02 11:23:36.946690] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c1123e0, pid=-100 lk-owner=640a6587ffff0000} [2018-02-02 11:23:36.946733] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on / [2018-02-02 11:23:36.946788] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan [2018-02-02 11:23:36.946948] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op [2018-02-02 11:23:36.947037] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0 [2018-02-02 11:23:39.222539] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-02-02 11:23:39.448907] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed [2018-02-02 11:23:39.493807] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2018-02-02 11:23:39.519128] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2018-02-02 11:28:57.534578] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-19277-2018/01/14-06:22:54:356470-test_vol-client-1-0-0 (version: 3.7.20) [root at f08n25glusterfs]# tail -n 100 etc-glusterfs-glusterd.vol.log [2018-02-02 11:21:56.188669] I [MSGID: 106540] [glusterd-utils.c:4409:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV3 successfully [2018-02-02 11:21:56.190907] I [MSGID: 106540] [glusterd-utils.c:4418:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV1 successfully [2018-02-02 11:21:56.193112] I [MSGID: 106540] [glusterd-utils.c:4427:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NFSV3 successfully [2018-02-02 11:21:56.195264] I [MSGID: 106540] [glusterd-utils.c:4436:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v4 successfully [2018-02-02 11:21:56.197410] I [MSGID: 106540] [glusterd-utils.c:4445:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v1 successfully [2018-02-02 11:21:56.199545] I [MSGID: 106540] [glusterd-utils.c:4454:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered ACL v3 successfully [2018-02-02 11:21:58.201432] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2018-02-02 11:21:58.201589] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2018-02-02 11:22:00.302648] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:22:32.998693] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:23:24.792241] W [common-utils.c:1685:gf_string2boolean] (-->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(gd_mgmt_v3_commit_fn+0x1f0) [0xffff789b7230] -->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0x290) [0xffff789816fc] -->/lib64/libglusterfs.so.0(gf_string2boolean+0x158) [0xffff82e676f0] ) 0-management: argument invalid [Invalid argument] [2018-02-02 11:23:24.967600] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol on port 49163 [2018-02-02 11:23:24.971147] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.080827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/22125620-6c93-40c3-9df0-962db4a347ac/test_vol on port 49164 [2018-02-02 11:23:25.082455] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.229069] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/223ff77c-2828-4f8e-ba38-21778227d135/test_vol on port 49165 [2018-02-02 11:23:25.230542] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.395235] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/22ed6b17-125a-4a17-b712-e7c9c4831105/test_vol on port 49166 [2018-02-02 11:23:25.396795] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.536292] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2401811b-351a-4083-933a-d990dd11ef06/test_vol on port 49167 [2018-02-02 11:23:25.537817] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.693177] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2fc12dde-54e1-4af2-a51e-70243bff72bd/test_vol on port 49168 [2018-02-02 11:23:25.694751] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.798878] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/37295c8b-fa3e-41c4-b10f-e7eba647bb35/test_vol on port 49169 [2018-02-02 11:23:25.800429] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:25.923224] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/4fe51006-a5bd-49c3-9eb7-ef2886184b46/test_vol on port 49170 [2018-02-02 11:23:25.924810] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:26.074144] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/58c1f8f7-7725-42d8-ab12-24b54476f48c/test_vol on port 49171 [2018-02-02 11:23:26.075499] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:26.181729] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/a08c683c-02bb-4395-bc19-8e1a0e03f62a/test_vol on port 49172 [2018-02-02 11:23:26.183043] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:26.332312] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/a2847ee4-d987-48ae-8fad-e17bd3e3298b/test_vol on port 49173 [2018-02-02 11:23:26.333655] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:26.435919] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/eca8cd11-d264-40ab-a384-4f8f56fc1e4f/test_vol on port 49174 [2018-02-02 11:23:26.437205] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2018-02-02 11:23:26.515934] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2018-02-02 11:23:26.516500] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2018-02-02 11:23:26.562053] W [socket.c:3045:socket_connect] 0-nfs: Ignore failed connection attempt on /var/run/gluster/e377f3a659bcefeef4591aaf11f965a3.socket, (No such file or directory) [2018-02-02 11:23:26.566840] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2018-02-02 11:23:26.574335] W [socket.c:3045:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4e1c4a574d86933847ffe01f15d99b7b.socket, (No such file or directory) [2018-02-02 11:23:26.574831] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2018-02-02 11:23:26.575059] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2018-02-02 11:23:26.579561] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2018-02-02 11:23:26.579698] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. [2018-02-02 11:23:28.897881] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2018-02-02 11:23:39.449912] W [dict.c:1452:dict_get_with_ref] (-->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(build_shd_graph+0x64) [0xffff78955628] -->/lib64/libglusterfs.so.0(dict_get_str_boolean+0x34) [0xffff82e41b30] -->/lib64/libglusterfs.so.0(+0x1f4ec) [0xffff82e3f4ec] ) 0-dict: dict OR key (graph-check) is NULL [Invalid argument] The message "I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 3 times between [2018-02-02 11:22:32.998693] and [2018-02-02 11:23:30.704852] [2018-02-02 11:23:39.491019] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2018-02-02 11:23:39.491341] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2018-02-02 11:23:40.395142] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:25:01.103516] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:26:31.489920] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:28:01.886988] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:29:32.202222] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:31:02.603808] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:32:33.004987] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:34:03.400885] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:35:33.734868] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:37:04.132280] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:38:34.524441] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:40:04.854724] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:41:35.233183] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:43:05.637442] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:44:35.954183] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:46:06.349265] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:47:36.747630] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:49:07.143301] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:50:37.537960] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:52:07.934435] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:53:38.277513] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:55:08.620253] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:56:38.924622] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:58:09.319604] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 11:59:39.712464] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:01:10.103154] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:02:40.500445] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:04:10.858003] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:05:41.184019] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:07:11.571314] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:08:41.966829] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:10:12.363095] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:11:42.757728] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:13:13.155581] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:14:43.553737] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:16:13.910490] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:17:44.306432] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:19:14.645538] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:20:44.962022] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:22:15.359878] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:23:45.739255] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:25:16.140107] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:26:46.539513] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:28:16.880023] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:29:47.257474] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:31:17.578904] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:32:47.898360] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:34:18.285139] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:35:48.677987] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:37:19.076224] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2018-02-02 12:38:49.425375] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180114/35ff7d36/attachment.html>