Gluster users - Nov 2012 - "layout is NULL", "Failed to get node-uuid for [...] and other errors during rebalancing in 3.3.1

I started rebalancing my volume after updating from 3.2.7 to 3.3.1. 
After a few hours, I noticed a large number of failures in the rebalance 
status:
>       Node Rebalanced-files size       scanned      failures         
> status
>  ---------      -----------   -----------   ----------- -----------   
> ------------
>  localhost                0        0Bytes 4288805             0        
> stopped
>       ml55            26275       206.2MB       4277101 14159        
> stopped
>       ml29                0        0Bytes 4288844             0        
> stopped
>       ml31                0        0Bytes 4288937             0        
> stopped
>       ml48                0        0Bytes 4288927             0        
> stopped
>       ml45            15041        50.8MB       4284304 41999        
> stopped
>       ml40            40690       413.3MB       4269721 1012        
> stopped
>       ml41                0        0Bytes 4288898             0        
> stopped
>       ml51            28558       212.7MB       4277442 32195        
> stopped
>       ml46                0        0Bytes 4288909             0        
> stopped
>       ml44                0        0Bytes 4288824             0        
> stopped
>       ml52                0        0Bytes 4288849             0        
> stopped
>       ml30            14252       183.7MB       4270711 25336        
> stopped
>       ml53            31431       354.9MB       4280450 31098        
> stopped
>       ml43            13773         2.7GB       4285256 28574        
> stopped
>       ml47            37618       241.3MB       4266889 24916        
> stopped
which prompted me to look at the rebalance log:
> [2012-11-30 11:06:12.533580] W 
> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-bigdata-client-12: remote 
> operation failed: File exists. Path: 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_269615212_b91ff3077e.t7 
> (00000000-0000-0000-0000-000000000000)
> [2012-11-30 11:06:12.533657] E [dht-common.c:1911:dht_getxattr] 
> 0-bigdata-dht: layout is NULL
> [2012-11-30 11:06:12.533702] E 
> [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-bigdata-dht: Failed to 
> get node-uuid for 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_269615212_b91ff3077e.t7
> [2012-11-30 11:06:12.545497] W 
> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-bigdata-client-13: remote 
> operation failed: File exists. Path: 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_217961761_965f9f192b.t7 
> (00000000-0000-0000-0000-000000000000)
> [2012-11-30 11:06:12.546039] W 
> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-bigdata-client-12: remote 
> operation failed: File exists. Path: 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_217961761_965f9f192b.t7 
> (00000000-0000-0000-0000-000000000000)
> [2012-11-30 11:06:12.546159] E [dht-common.c:1911:dht_getxattr] 
> 0-bigdata-dht: layout is NULL
> [2012-11-30 11:06:12.546199] E 
> [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-bigdata-dht: Failed to 
> get node-uuid for 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_217961761_965f9f192b.t7
> [2012-11-30 11:06:12.617940] W 
> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-bigdata-client-12: remote 
> operation failed: File exists. Path: 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_211665292_59a24211c3.t7 
> (00000000-0000-0000-0000-000000000000)
> [2012-11-30 11:06:12.618024] W 
> [client3_1-fops.c:258:client3_1_mknod_cbk] 0-bigdata-client-13: remote 
> operation failed: File exists. Path: 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_211665292_59a24211c3.t7 
> (00000000-0000-0000-0000-000000000000)
> [2012-11-30 11:06:12.618150] E [dht-common.c:1911:dht_getxattr] 
> 0-bigdata-dht: layout is NULL
> [2012-11-30 11:06:12.618189] E 
> [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-bigdata-dht: Failed to 
> get node-uuid for 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_211665292_59a24211c3.t7
> [2012-11-30 11:06:12.620798] I 
> [dht-common.c:954:dht_lookup_everywhere_cbk] 0-bigdata-dht: deleting 
> stale linkfile 
> /foo/data/onemil/dataset/bar/f8old/baz/85/m_85_282643649_15d4108d95.t7 
> on bigdata-replicate-6
[...] (at this point, I stopped rebalancing, and got the following in 
the logs)
> [2012-11-30 11:06:33.152153] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil/dataset/bar/f8old/baz/85
> [2012-11-30 11:06:33.153628] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil/dataset/bar/f8old/baz
> [2012-11-30 11:06:33.154641] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil/dataset/bar/f8old
> [2012-11-30 11:06:33.155602] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil/dataset/bar
> [2012-11-30 11:06:33.156552] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil/dataset
> [2012-11-30 11:06:33.157538] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data/onemil
> [2012-11-30 11:06:33.158526] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo/data
> [2012-11-30 11:06:33.159459] E 
> [dht-rebalance.c:1374:gf_defrag_fix_layout] 0-bigdata-dht: Fix layout 
> failed for /foo
> [2012-11-30 11:06:33.160496] I 
> [dht-rebalance.c:1626:gf_defrag_status_get] 0-glusterfs: Rebalance is 
> stopped
> [2012-11-30 11:06:33.160518] I 
> [dht-rebalance.c:1629:gf_defrag_status_get] 0-glusterfs: Files 
> migrated: 14252, size: 192620657, lookups: 4270711, failures: 25336
> [2012-11-30 11:06:33.173344] W [glusterfsd.c:831:cleanup_and_exit] 
> (-->/lib64/libc.so.6(clone+0x6d) [0x3d676e811d] 
> (-->/lib64/libpthread.so.0() [0x3d68207851] 
> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: 
> received signum (15), shutting down

This kind of error kept appearing many times per second. I cancelled the 
rebalancing operation just in case it was doing anything bad. There 
doesn't seem to be anything weird in the system logs of the bricks that 
hold the files mentioned in the errors. The files are still accessible 
through my mounted volume.

Any idea what might be wrong?

Thanks,

Pierre



volume info:
> Volume Name: bigdata
> Type: Distributed-Replicate
> Volume ID: 56498956-7b4b-4ee3-9d2b-4c8cfce26051
> Status: Started
> Number of Bricks: 20 x 2 = 40
> Transport-type: tcp
> Bricks:
> Brick1: ml43:/mnt/localb
> Brick2: ml44:/mnt/localb
> Brick3: ml43:/mnt/localc
> Brick4: ml44:/mnt/localc
> Brick5: ml45:/mnt/localb
> Brick6: ml46:/mnt/localb
> Brick7: ml45:/mnt/localc
> Brick8: ml46:/mnt/localc
> Brick9: ml47:/mnt/localb
> Brick10: ml48:/mnt/localb
> Brick11: ml47:/mnt/localc
> Brick12: ml48:/mnt/localc
> Brick13: ml45:/mnt/locald
> Brick14: ml46:/mnt/locald
> Brick15: ml47:/mnt/locald
> Brick16: ml48:/mnt/locald
> Brick17: ml51:/mnt/localb
> Brick18: ml52:/mnt/localb
> Brick19: ml51:/mnt/localc
> Brick20: ml52:/mnt/localc
> Brick21: ml51:/mnt/locald
> Brick22: ml52:/mnt/locald
> Brick23: ml53:/mnt/locald
> Brick24: ml54:/mnt/locald
> Brick25: ml53:/mnt/localc
> Brick26: ml54:/mnt/localc
> Brick27: ml53:/mnt/localb
> Brick28: ml54:/mnt/localb
> Brick29: ml55:/mnt/localb
> Brick30: ml29:/mnt/localb
> Brick31: ml55:/mnt/localc
> Brick32: ml29:/mnt/localc
> Brick33: ml30:/mnt/localc
> Brick34: ml31:/mnt/localc
> Brick35: ml30:/mnt/localb
> Brick36: ml31:/mnt/localb
> Brick37: ml40:/mnt/localb
> Brick38: ml41:/mnt/localb
> Brick39: ml40:/mnt/localc
> Brick40: ml41:/mnt/localc
> Options Reconfigured:
> performance.quick-read: on
> nfs.disable: on
> nfs.register-with-portmap: OFF

volume status:
> Status of volume: bigdata
> Gluster process                                         Port Online  Pid
>
------------------------------------------------------------------------------
> Brick ml43:/mnt/localb                                  24012 Y       2694
> Brick ml44:/mnt/localb                                  24012 Y       
> 20374
> Brick ml43:/mnt/localc                                  24013 Y       2699
> Brick ml44:/mnt/localc                                  24013 Y       
> 20379
> Brick ml45:/mnt/localb                                  24012 Y       3147
> Brick ml46:/mnt/localb                                  24012 Y       
> 25789
> Brick ml45:/mnt/localc                                  24013 Y       3152
> Brick ml46:/mnt/localc                                  24013 Y       
> 25794
> Brick ml47:/mnt/localb                                  24012 Y       3181
> Brick ml48:/mnt/localb                                  24012 Y       4852
> Brick ml47:/mnt/localc                                  24013 Y       3186
> Brick ml48:/mnt/localc                                  24013 Y       4857
> Brick ml45:/mnt/locald                                  24014 Y       3157
> Brick ml46:/mnt/locald                                  24014 Y       
> 25799
> Brick ml47:/mnt/locald                                  24014 Y       3191
> Brick ml48:/mnt/locald                                  24014 Y       4862
> Brick ml51:/mnt/localb                                  24009 Y       
> 30251
> Brick ml52:/mnt/localb                                  24012 Y       
> 28541
> Brick ml51:/mnt/localc                                  24010 Y       
> 30256
> Brick ml52:/mnt/localc                                  24013 Y       
> 28546
> Brick ml51:/mnt/locald                                  24011 Y       
> 30261
> Brick ml52:/mnt/locald                                  24014 Y       
> 28551
> Brick ml53:/mnt/locald                                  24012 Y       9229
> Brick ml54:/mnt/locald                                  24012 Y       9341
> Brick ml53:/mnt/localc                                  24013 Y       9234
> Brick ml54:/mnt/localc                                  24013 Y       9346
> Brick ml53:/mnt/localb                                  24014 Y       9239
> Brick ml54:/mnt/localb                                  24014 Y       9351
> Brick ml55:/mnt/localb                                  24012 Y       
> 30904
> Brick ml29:/mnt/localb                                  24012 Y       
> 29233
> Brick ml55:/mnt/localc                                  24013 Y       
> 30909
> Brick ml29:/mnt/localc                                  24013 Y       
> 29238
> Brick ml30:/mnt/localc                                  24012 Y       6800
> Brick ml31:/mnt/localc                                  N/A Y       22000
> Brick ml30:/mnt/localb                                  24013 Y       6805
> Brick ml31:/mnt/localb                                  N/A Y       22005
> Brick ml40:/mnt/localb                                  24012 Y       
> 26700
> Brick ml41:/mnt/localb                                  24012 Y       
> 25762
> Brick ml40:/mnt/localc                                  24013 Y       
> 26705
> Brick ml41:/mnt/localc                                  24013 Y       
> 25767
> Self-heal Daemon on localhost                           N/A Y       20392
> Self-heal Daemon on ml55                                N/A Y       30922
> Self-heal Daemon on ml54                                N/A Y       9365
> Self-heal Daemon on ml52                                N/A Y       28565
> Self-heal Daemon on ml29                                N/A Y       29253
> Self-heal Daemon on ml30                                N/A Y       6818
> Self-heal Daemon on ml43                                N/A Y       2712
> Self-heal Daemon on ml47                                N/A Y       3205
> Self-heal Daemon on ml46                                N/A Y       25813
> Self-heal Daemon on ml40                                N/A Y       26717
> Self-heal Daemon on ml31                                N/A Y       22038
> Self-heal Daemon on ml48                                N/A Y       4876
> Self-heal Daemon on ml45                                N/A Y       3171
> Self-heal Daemon on ml51                                N/A Y       30274
> Self-heal Daemon on ml41                                N/A Y       25779
> Self-heal Daemon on ml53                                N/A Y       9253
peer status:
> Number of Peers: 15
>
> Hostname: ml52
> Uuid: 4de42f67-4cca-4d28-8600-9018172563ba
> State: Peer in Cluster (Connected)
>
> Hostname: ml41
> Uuid: b404851f-dfd5-4746-a3bd-81bb0d888009
> State: Peer in Cluster (Connected)
>
> Hostname: ml46
> Uuid: af74d39b-09d6-47ba-9c3b-72d993dca4ce
> State: Peer in Cluster (Connected)
>
> Hostname: ml54
> Uuid: c55580fa-2c9d-493d-b9d1-3bce016c8b29
> State: Peer in Cluster (Connected)
>
> Hostname: ml51
> Uuid: 5491b6dc-0f96-43d9-95d9-a41018a8542c
> State: Peer in Cluster (Connected)
>
> Hostname: ml48
> Uuid: efd79145-bfd9-4eea-b7a7-50be18d9ffe0
> State: Peer in Cluster (Connected)
>
> Hostname: ml43
> Uuid: a9044e9a-39e1-4907-8921-43da870b7f31
> State: Peer in Cluster (Connected)
>
> Hostname: ml45
> Uuid: 0eebbceb-8f62-4c55-8160-41348f90e191
> State: Peer in Cluster (Connected)
>
> Hostname: ml47
> Uuid: e831092d-b196-46ec-947d-a5635e8fbd1e
> State: Peer in Cluster (Connected)
>
> Hostname: ml30
> Uuid: e56b4c57-a058-4464-a1e6-c4676ebf00cc
> State: Peer in Cluster (Connected)
>
> Hostname: ml40
> Uuid: ffcc06ae-100a-4fa2-888e-803a41ae946c
> State: Peer in Cluster (Connected)
>
> Hostname: ml55
> Uuid: 366339ed-52e5-4722-a1b3-e3bb1c49ea4f
> State: Peer in Cluster (Connected)
>
> Hostname: ml31
> Uuid: 699019f6-2f4a-45cb-bfa4-f209745f8a6d
> State: Peer in Cluster (Connected)
>
> Hostname: ml29
> Uuid: 58aa8a16-5d2b-4c06-8f06-2fd0f7fc5a37
> State: Peer in Cluster (Connected)
>
> Hostname: ml53
> Uuid: 1dc6ee08-c606-4755-8756-b553f66efa88
> State: Peer in Cluster (Connected)
gluster version:
> glusterfs 3.3.1 built on Oct 11 2012 21:49:37
rpms:
> glusterfs.x86_64 3.3.1-1.el6                @glusterfs-epel
> glusterfs-debuginfo.x86_64            3.3.1-1.el6 @glusterfs-epel
> glusterfs-fuse.x86_64                 3.3.1-1.el6 @glusterfs-epel
> glusterfs-rdma.x86_64                 3.3.1-1.el6 @glusterfs-epel
> glusterfs-server.x86_64               3.3.1-1.el6 @glusterfs-epel
kernel:
> Linux 2.6.32-131.17.1.el6.x86_64 #1 SMP Wed Oct 5 17:19:54 CDT 2011 
> x86_64 x86_64 x86_64 GNU/Linux
OS: Scientific Linux 6.1 (this is based on CentOS)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121130/a09f841b/attachment.html>

On 11/30/2012 11:47, Pierre-Francois Laquerre wrote:
> <snip>
To add to this, I just found some more interesting errors in the client 
logs:

[2012-11-30 11:46:29.317376] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-4: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:29.317519] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-5: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:29.317606] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-7: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:29.317828] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-4: background  meta-data self-heal failed on /
[2012-11-30 11:46:29.317909] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-5: background  meta-data self-heal failed on /
[2012-11-30 11:46:29.318019] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-7: background  meta-data self-heal failed on /
[2012-11-30 11:46:29.336294] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-15: Unable to self-heal permissions/ownership of 
'/foo/data' (possible split-brain). Please fix the file on all backend 
volumes
[2012-11-30 11:46:29.336490] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-14: Unable to self-heal permissions/ownership of 
'/foo/data' (possible split-brain). Please fix the file on all backend 
volumes
[2012-11-30 11:46:29.336731] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-15: background  meta-data self-heal failed on /foo/data
[2012-11-30 11:46:29.336842] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-14: background  meta-data self-heal failed on /foo/data
[2012-11-30 11:46:54.580455] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-4: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:54.581099] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-4: background  meta-data self-heal failed on /
[2012-11-30 11:46:54.581263] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-7: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:54.581349] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-5: Unable to self-heal permissions/ownership of '/' 
(possible split-brain). Please fix the file on all backend volumes
[2012-11-30 11:46:54.581575] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-7: background  meta-data self-heal failed on /
[2012-11-30 11:46:54.581666] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-5: background  meta-data self-heal failed on /
[2012-11-30 11:46:54.593012] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-15: Unable to self-heal permissions/ownership of 
'/foo/data' (possible split-brain). Please fix the file on all backend 
volumes
[2012-11-30 11:46:54.593098] E 
[afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 
2-bigdata-replicate-14: Unable to self-heal permissions/ownership of 
'/foo/data' (possible split-brain). Please fix the file on all backend 
volumes
[2012-11-30 11:46:54.593338] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-15: background  meta-data self-heal failed on /foo/data
[2012-11-30 11:46:54.593425] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
2-bigdata-replicate-14: background  meta-data self-heal failed on /foo/data


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121130/fb368b1d/attachment.html>

On Fri, 2012-11-30 at 11:47 -0500, Pierre-Francois Laquerre wrote:
> 
> This kind of error kept appearing many times per second. I cancelled
> the rebalancing operation just in case it was doing anything bad.
> There doesn't seem to be anything weird in the system logs of the
> bricks that hold the files mentioned in the errors. The files are
> still accessible through my mounted volume.
> 
> Any idea what might be wrong?
Yes, 3.3.1 is stupid, damned version. it should never be used! It's
contain huge bugs. The background heal feature is messed working.
-- 
M.At?f CEYLAN
Yurdum Yaz?l?m
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121130/f7e87e1f/attachment.html>