Hello i have some strange problem that happens on one of the servers with glusterfs. Configuration: -- gluster volume info -- Volume Name: gisdata Type: Distribute Volume ID: a49bcb69-fcc5-4f39-b556-e23e29da38e7 Status: Started Number of Bricks: 7 Transport-type: tcp Bricks: Brick1: 10.1.0.4:/export/sdc1/brick Brick2: 10.1.0.5:/export/sdc1/brick Brick3: 10.1.0.6:/export/sdc1/brick Brick4: 10.1.0.7:/export/sdc1/brick Brick5: 10.1.0.9:/export/sdc1/brick Brick6: 10.1.0.10:/export/sdc1/brick Brick7: 10.1.0.12:/export/sdc1/brick Options Reconfigured: server.statedump-path: /tmp performance.cache-size: 512MB nfs.drc: off System mount to master server as 10.1.0.4:/gisdata on /usr/local/mountpoint type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) and its share by samba to the client who manage files on distribute filesystem. Client tryes to delete directory (Data/Projects/DELIVERY/D39) but it fails with error: Can't delete, directory is not empty. When i inspect that directory on the Bricks i found it totaly broken ls -lah /export/sdc1/brick/Data/Projects/DELIVERY/D39/ ls: cannot access /export/sdc1/brick/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO: No such file or directory total 16K drwxr-xr-x 3 root 500 8.0K Dec 1 10:52 . drwxr-xr-x 40 root 500 4.0K Nov 30 22:43 .. ?????????? ? ? ? ? ? 494-11-23-1_WD_PO When i recreate that directory it looks fine but has doubled: ls -lai /export/sdc1/brick/Data/Projects/DELIVERY/D39/ total 16 30075838 drwxr-xr-x 4 root 500 8192 Dec 1 10:56 . 9185992 drwxr-xr-x 40 root 500 4096 Nov 30 22:43 .. 2147631741 drwxr-xr-x 2 root root 6 Dec 1 10:56 494-11-23-1_WD_PO 2147631741 drwxr-xr-x 2 root root 6 Dec 1 10:56 494-11-23-1_WD_PO Last logs: /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.352895] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.355764] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.358448] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.381921] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.843840] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.846111] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.848384] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.857213] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.678051] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.680209] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.682785] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.697692] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.880597] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.885190] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.889059] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.905035] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.985591] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.990133] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.995346] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:49.012616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.278130] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO /var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.283321] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01 /var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.286403] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH /var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.305513] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES /var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 09:52:26.319979] E [dht-rebalance.c:1496:gf_defrag_fix_layout] 0-gisdata-dht: /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO lookup failed /var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.172294] I [dht-layout.c:640:dht_layout_normalize] 0-gisdata-dht: found anomalies in /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO. holes=1 overlaps=0 /var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.173616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO It happens on server 10.1.0.9. Any ideas how to fix that? -- _________________________ - tetsucceed -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161201/3ff04dca/attachment.html>