Hi, I had a 2 node distributed cluster running on 3.1.1 and I added 2 more nodes. I then ran a rebalance on the cluster. Now I am getting permission denied errors and I see the following in the client logs: [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [root at qe-loader1 glusterfs]# tail -100 mnt-qe-filer01.log [2011-02-24 09:32:50.844211] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.845312] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0 [2011-02-24 09:32:50.846417] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-0; inode layout - 0 - 2147483646; disk layout - 2147483646 - 3221225468 [2011-02-24 09:32:50.846434] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.846561] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-1; inode layout - 2147483647 - 4294967295; disk layout - 3221225469 - 4294967295 [2011-02-24 09:32:50.846577] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.847650] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0 [2011-02-24 09:39:08.159175] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /mnt/qe-filer01 [2011-02-24 09:39:08.160180] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down [2011-02-24 09:39:14.184554] W [io-stats.c:1644:init] loader: dangling volume. check volfile [2011-02-24 09:39:14.184599] W [dict.c:1204:data_to_str] dict: @data=(nil) [2011-02-24 09:39:14.184614] W [dict.c:1204:data_to_str] dict: @data=(nil) Given volfile: +------------------------------------------------------------------------------+ 1: volume loader-client-0 2: type protocol/client 3: option remote-host qe-filer03 4: option remote-subvolume /data1 5: option transport-type tcp 6: end-volume 7: 8: volume loader-client-1 9: type protocol/client 10: option remote-host qe-filer04 11: option remote-subvolume /data1 12: option transport-type tcp 13: end-volume 14: 15: volume loader-client-2 16: type protocol/client 17: option remote-host 172.24.8.25 18: option remote-subvolume /data1 19: option transport-type tcp 20: end-volume 21: 22: volume loader-client-3 23: type protocol/client 24: option remote-host qe-filer02 25: option remote-subvolume /data1 26: option transport-type tcp 27: end-volume 28: 29: volume loader-dht 30: type cluster/distribute 31: subvolumes loader-client-0 loader-client-1 loader-client-2 loader-client-3 32: end-volume 33: 34: volume loader-write-behind 35: type performance/write-behind 36: subvolumes loader-dht 37: end-volume 38: 39: volume loader-read-ahead 40: type performance/read-ahead 41: subvolumes loader-write-behind 42: end-volume 43: 44: volume loader-io-cache 45: type performance/io-cache 46: subvolumes loader-read-ahead 47: end-volume 48: 49: volume loader-quick-read 50: type performance/quick-read 51: subvolumes loader-io-cache 52: end-volume 53: 54: volume loader-stat-prefetch 55: type performance/stat-prefetch 56: subvolumes loader-quick-read 57: end-volume 58: 59: volume loader 60: type debug/io-stats 61: subvolumes loader-stat-prefetch 62: end-volume +------------------------------------------------------------------------------+ [2011-02-24 09:39:18.132954] I [client-handshake.c:993:select_server_supported_programs] loader-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.136077] I [client-handshake.c:829:client_setvolume_cbk] loader-client-2: Connected to 172.24.8.25:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.136498] I [client-handshake.c:993:select_server_supported_programs] loader-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.137083] I [client-handshake.c:829:client_setvolume_cbk] loader-client-1: Connected to 172.24.8.222:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.140002] I [client-handshake.c:993:select_server_supported_programs] loader-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.140599] I [client-handshake.c:829:client_setvolume_cbk] loader-client-0: Connected to 172.24.8.221:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.144060] I [client-handshake.c:993:select_server_supported_programs] loader-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.144689] I [client-handshake.c:829:client_setvolume_cbk] loader-client-3: Connected to 172.24.8.220:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.150335] I [fuse-bridge.c:2817:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2011-02-24 09:54:12.904354] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 4094: UNLINK() / c9.lock => -1 (Permission denied) [2011-02-24 09:54:34.716535] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:54:36.470275] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:24.588912] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:26.180690] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:57.816431] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:59.242725] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:56:54.92112] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23558: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:57:01.792819] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23564: /Cloud9c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:57:31.812728] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23570: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:58:12.738834] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23582: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) I am running the rebalance again to see if it will fix the layout issue. Any ideas why I'm seeing these errors after adding new nodes? Regards, John Lao
Pranith Kumar. Karampuri
2011-Feb-25 06:11 UTC
[Gluster-users] Experiencing errors after adding new nodes
Rebalance and EPERM, looks like you are hitting http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2369 Pranith. ----- Original Message ----- From: "John Lao" <jlao at cloud9analytics.com> To: gluster-users at gluster.org Sent: Thursday, February 24, 2011 11:55:52 PM Subject: [Gluster-users] Experiencing errors after adding new nodes Hi, I had a 2 node distributed cluster running on 3.1.1 and I added 2 more nodes. I then ran a rebalance on the cluster. Now I am getting permission denied errors and I see the following in the client logs: [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [root at qe-loader1 glusterfs]# tail -100 mnt-qe-filer01.log [2011-02-24 09:32:50.844211] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.845312] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0 [2011-02-24 09:32:50.846417] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-0; inode layout - 0 - 2147483646; disk layout - 2147483646 - 3221225468 [2011-02-24 09:32:50.846434] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.846561] I [dht-layout.c:689:dht_layout_dir_mismatch] loader-dht: subvol: loader-client-1; inode layout - 2147483647 - 4294967295; disk layout - 3221225469 - 4294967295 [2011-02-24 09:32:50.846577] I [dht-common.c:415:dht_revalidate_cbk] loader-dht: mismatching layouts for / [2011-02-24 09:32:50.847650] I [dht-layout.c:588:dht_layout_normalize] loader-dht: found anomalies in /. holes=1 overlaps=0 [2011-02-24 09:39:08.159175] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /mnt/qe-filer01 [2011-02-24 09:39:08.160180] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down [2011-02-24 09:39:14.184554] W [io-stats.c:1644:init] loader: dangling volume. check volfile [2011-02-24 09:39:14.184599] W [dict.c:1204:data_to_str] dict: @data=(nil) [2011-02-24 09:39:14.184614] W [dict.c:1204:data_to_str] dict: @data=(nil) Given volfile: +------------------------------------------------------------------------------+ 1: volume loader-client-0 2: type protocol/client 3: option remote-host qe-filer03 4: option remote-subvolume /data1 5: option transport-type tcp 6: end-volume 7: 8: volume loader-client-1 9: type protocol/client 10: option remote-host qe-filer04 11: option remote-subvolume /data1 12: option transport-type tcp 13: end-volume 14: 15: volume loader-client-2 16: type protocol/client 17: option remote-host 172.24.8.25 18: option remote-subvolume /data1 19: option transport-type tcp 20: end-volume 21: 22: volume loader-client-3 23: type protocol/client 24: option remote-host qe-filer02 25: option remote-subvolume /data1 26: option transport-type tcp 27: end-volume 28: 29: volume loader-dht 30: type cluster/distribute 31: subvolumes loader-client-0 loader-client-1 loader-client-2 loader-client-3 32: end-volume 33: 34: volume loader-write-behind 35: type performance/write-behind 36: subvolumes loader-dht 37: end-volume 38: 39: volume loader-read-ahead 40: type performance/read-ahead 41: subvolumes loader-write-behind 42: end-volume 43: 44: volume loader-io-cache 45: type performance/io-cache 46: subvolumes loader-read-ahead 47: end-volume 48: 49: volume loader-quick-read 50: type performance/quick-read 51: subvolumes loader-io-cache 52: end-volume 53: 54: volume loader-stat-prefetch 55: type performance/stat-prefetch 56: subvolumes loader-quick-read 57: end-volume 58: 59: volume loader 60: type debug/io-stats 61: subvolumes loader-stat-prefetch 62: end-volume +------------------------------------------------------------------------------+ [2011-02-24 09:39:18.132954] I [client-handshake.c:993:select_server_supported_programs] loader-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.136077] I [client-handshake.c:829:client_setvolume_cbk] loader-client-2: Connected to 172.24.8.25:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.136498] I [client-handshake.c:993:select_server_supported_programs] loader-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.137083] I [client-handshake.c:829:client_setvolume_cbk] loader-client-1: Connected to 172.24.8.222:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.140002] I [client-handshake.c:993:select_server_supported_programs] loader-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.140599] I [client-handshake.c:829:client_setvolume_cbk] loader-client-0: Connected to 172.24.8.221:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.144060] I [client-handshake.c:993:select_server_supported_programs] loader-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-02-24 09:39:18.144689] I [client-handshake.c:829:client_setvolume_cbk] loader-client-3: Connected to 172.24.8.220:24009, attached to remote volume '/data1'. [2011-02-24 09:39:18.150335] I [fuse-bridge.c:2817:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2011-02-24 09:54:12.904354] W [fuse-bridge.c:888:fuse_unlink_cbk] glusterfs-fuse: 4094: UNLINK() / c9.lock => -1 (Permission denied) [2011-02-24 09:54:34.716535] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:54:36.470275] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:24.588912] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:26.180690] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:57.816431] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:55:59.242725] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:56:54.92112] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23558: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:57:01.792819] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23564: /Cloud9c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:57:31.812728] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23570: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:58:12.738834] W [fuse-bridge.c:1510:fuse_create_cbk] glusterfs-fuse: 23582: / c9_refresh.lock => -1 (Permission denied) [2011-02-24 09:59:10.210166] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) [2011-02-24 09:59:11.851656] I [dht-common.c:369:dht_revalidate_cbk] loader-dht: subvolume loader-client-3 returned -1 (Invalid argument) I am running the rebalance again to see if it will fix the layout issue. Any ideas why I'm seeing these errors after adding new nodes? Regards, John Lao _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users