Benedikt Fraunhofer
2013-Jul-08 07:43 UTC
[Gluster-users] fuse 3.3.1 fails/crashes on flush on Distributed-Striped-Replicate volume
Hello List, before filing a bug I wanted to check with the community if this is a known issue. I'm running gluster 3.3.1 from the semiosis PPA on ubuntu 12.04. I created a 2x2x2 volume: gluster> volume info Volume Name: gv0-s2-r2 Type: Distributed-Striped-Replicate Volume ID: 8e1f1e86-63c0-4c27-9c90-fef92bdd0ca1 Status: Started Number of Bricks: 2 x 2 x 2 = 8 Transport-type: tcp Bricks: Brick1: 172.31.32.33:/gluster/brick0 Brick2: 172.31.32.34:/gluster/brick0 Brick3: 172.31.32.37:/gluster/brick0 Brick4: 172.31.32.38:/gluster/brick0 Brick5: 172.31.32.39:/gluster/brick0 Brick6: 172.31.32.40:/gluster/brick0 Brick7: 172.31.32.42:/gluster/brick0 Brick8: 172.31.32.47:/gluster/brick0 which mounts fine, but it started to smell fishy when vim wasn't able to write its swapfile, but I ignored that. I hacked my stress-script together which makes the fuse-mount instantly unuseable. The script just creates random file- and directory-names and tries to write data to these: Essentially it does something like: mkdir /mnt/a/C/C2/C2s/ touch /mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo head -c 12837 /dev/zero > /mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo which results in: stress.sh: line 28: /mnt/a/y/yK/yK2/yK2ujHHPRJYukCaqFhYkAN5MVDJxoAmTu89nM5auLA: Software caused connection abort the following calls result in: mkdir: cannot create directory `/mnt/a/l': Transport endpoint is not connected because the mount is gone. I thought I hit the ext4 64bit-hash bug, reformatted to xfs, but same game. note that this only happens when done quickly; if you wait between directory creation, touch and write it works: root at gluster-71:/mnt/a# mkdir -p a/b/c/d root at gluster-71:/mnt/a# touch a/b/c/d/e root at gluster-71:/mnt/a# head -c 32 /dev/zero > a/b/c/d/e root at gluster-71:/mnt/a# mkdir -p b/c/d/; touch b/c/d/e; head -c 32 /dev/zero >b/c/d/e -bash: b/c/d/e: Software caused connection abort logging says: [2013-07-05 15:24:05.764677] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.19 [2013-07-05 15:24:05.765679] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-0: added root inode [2013-07-05 15:24:05.765747] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-1: added root inode [2013-07-05 15:24:05.767220] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-3: added root inode [2013-07-05 15:24:05.767304] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-2: added root inode [2013-07-05 15:24:09.709367] I [dict.c:317:dict_get] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c) [0x7fd39d7262ac] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb) [0x7fd39d507bab] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90) [0x7fd39d510150]))) 0-dict: !this || key=trusted.gv0-s2-r2-stripe-1.stripe-size [2013-07-05 15:24:09.709446] E [stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-1: Failed to get stripe-size [2013-07-05 15:24:09.714424] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 17: FLUSH() ERR => -1 (Invalid argument) [2013-07-05 15:24:57.247208] I [dict.c:317:dict_get] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c) [0x7fd39d7262ac] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb) [0x7fd39d507bab] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90) [0x7fd39d510150]))) 0-dict: !this || key=trusted.gv0-s2-r2-stripe-0.stripe-size [2013-07-05 15:24:57.247436] E [stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-0: Failed to get stripe-size pending frames: frame : type(1) op(TRUNCATE) frame : type(1) op(OPEN) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) patchset: git://git.gluster.com/glusterfs.git signal received: 8 time of crash: 2013-07-05 15:24:57 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.1 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7fd3a1d224a0] /usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_truncate+0x28c)[0x7fd39d50287c] /usr/lib/glusterfs/3.3.1/xlator/cluster/distribute.so(dht_truncate+0x172)[0x7fd39d2dd8c2] /usr/lib/glusterfs/3.3.1/xlator/performance/write-behind.so(wb_truncate+0x49e)[0x7fd39d0a769e] /usr/lib/glusterfs/3.3.1/xlator/performance/read-ahead.so(ra_truncate+0x1b2)[0x7fd39ce96302] /usr/lib/glusterfs/3.3.1/xlator/performance/io-cache.so(ioc_truncate+0x14c)[0x7fd39cc84aec] /usr/lib/libglusterfs.so.0(default_truncate+0x124)[0x7fd3a271c584] /usr/lib/glusterfs/3.3.1/xlator/performance/md-cache.so(mdc_truncate+0x13a)[0x7fd39c8625fa] /usr/lib/glusterfs/3.3.1/xlator/debug/io-stats.so(io_stats_truncate+0x14f)[0x7fd39c64bb3f] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x17b7f)[0x7fd3a0001b7f] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bd6)[0x7fd39fff0bd6] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6ae2)[0x7fd39fff0ae2] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bce)[0x7fd39fff0bce] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_continue+0x41)[0x7fd39fff0851] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_inode+0x25)[0x7fd39fff0975] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6b05)[0x7fd39fff0b05] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bae)[0x7fd39fff0bae] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_and_resume+0x28)[0x7fd39fff1128] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x1af74)[0x7fd3a0004f74] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd3a20b2e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd3a1ddfccd] I tried to read all the bugs which include "Failed to get stripe-size" but they don't include anything fuse-related; anyhow it reads like https://bugzilla.redhat.com/show_bug.cgi?id=861423 which http://review.gluster.org/3904 should be solving? Thx in advance Benedikt.
Kushnir, Michael (NIH/NLM/LHC) [C]
2013-Jul-10 18:38 UTC
[Gluster-users] fuse 3.3.1 fails/crashes on flush on Distributed-Striped-Replicate volume
I had the same problem with striped-replicated. https://bugzilla.redhat.com/show_bug.cgi?id=861423 Best, Michael -----Original Message----- From: Benedikt Fraunhofer [mailto:benedikt.fraunhofer.l.gluster.fxy-3zz-w8t at traced.net] Sent: Monday, July 08, 2013 3:43 AM To: Gluster-users at gluster.org Subject: [Gluster-users] fuse 3.3.1 fails/crashes on flush on Distributed-Striped-Replicate volume Hello List, before filing a bug I wanted to check with the community if this is a known issue. I'm running gluster 3.3.1 from the semiosis PPA on ubuntu 12.04. I created a 2x2x2 volume: gluster> volume info Volume Name: gv0-s2-r2 Type: Distributed-Striped-Replicate Volume ID: 8e1f1e86-63c0-4c27-9c90-fef92bdd0ca1 Status: Started Number of Bricks: 2 x 2 x 2 = 8 Transport-type: tcp Bricks: Brick1: 172.31.32.33:/gluster/brick0 Brick2: 172.31.32.34:/gluster/brick0 Brick3: 172.31.32.37:/gluster/brick0 Brick4: 172.31.32.38:/gluster/brick0 Brick5: 172.31.32.39:/gluster/brick0 Brick6: 172.31.32.40:/gluster/brick0 Brick7: 172.31.32.42:/gluster/brick0 Brick8: 172.31.32.47:/gluster/brick0 which mounts fine, but it started to smell fishy when vim wasn't able to write its swapfile, but I ignored that. I hacked my stress-script together which makes the fuse-mount instantly unuseable. The script just creates random file- and directory-names and tries to write data to these: Essentially it does something like: mkdir /mnt/a/C/C2/C2s/ touch /mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo head -c 12837 /dev/zero > /mnt/a/C/C2/C2s/C2syqS30tsA5sctL2C8j22zd04VWsZDj3fZSWCK8Uo which results in: stress.sh: line 28: /mnt/a/y/yK/yK2/yK2ujHHPRJYukCaqFhYkAN5MVDJxoAmTu89nM5auLA: Software caused connection abort the following calls result in: mkdir: cannot create directory `/mnt/a/l': Transport endpoint is not connected because the mount is gone. I thought I hit the ext4 64bit-hash bug, reformatted to xfs, but same game. note that this only happens when done quickly; if you wait between directory creation, touch and write it works: root at gluster-71:/mnt/a# mkdir -p a/b/c/d root at gluster-71:/mnt/a# touch a/b/c/d/e root at gluster-71:/mnt/a# head -c 32 /dev/zero > a/b/c/d/e root at gluster-71:/mnt/a# mkdir -p b/c/d/; touch b/c/d/e; head -c 32 /dev/zero >b/c/d/e -bash: b/c/d/e: Software caused connection abort logging says: [2013-07-05 15:24:05.764677] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.19 [2013-07-05 15:24:05.765679] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-0: added root inode [2013-07-05 15:24:05.765747] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-1: added root inode [2013-07-05 15:24:05.767220] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-3: added root inode [2013-07-05 15:24:05.767304] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-gv0-s2-r2-replicate-2: added root inode [2013-07-05 15:24:09.709367] I [dict.c:317:dict_get] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c) [0x7fd39d7262ac] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb) [0x7fd39d507bab] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90) [0x7fd39d510150]))) 0-dict: !this || key=trusted.gv0-s2-r2-stripe-1.stripe-size [2013-07-05 15:24:09.709446] E [stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-1: Failed to get stripe-size [2013-07-05 15:24:09.714424] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 17: FLUSH() ERR => -1 (Invalid argument) [2013-07-05 15:24:57.247208] I [dict.c:317:dict_get] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_create_unwind+0x13c) [0x7fd39d7262ac] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_create_cbk+0x5fb) [0x7fd39d507bab] (-->/usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_ctx_handle+0x90) [0x7fd39d510150]))) 0-dict: !this || key=trusted.gv0-s2-r2-stripe-0.stripe-size [2013-07-05 15:24:57.247436] E [stripe-helpers.c:268:stripe_ctx_handle] 0-gv0-s2-r2-stripe-0: Failed to get stripe-size pending frames: frame : type(1) op(TRUNCATE) frame : type(1) op(OPEN) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) frame : type(1) op(FLUSH) patchset: git://git.gluster.com/glusterfs.git signal received: 8 time of crash: 2013-07-05 15:24:57 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.1 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0)[0x7fd3a1d224a0] /usr/lib/glusterfs/3.3.1/xlator/cluster/stripe.so(stripe_truncate+0x28c)[0x7fd39d50287c] /usr/lib/glusterfs/3.3.1/xlator/cluster/distribute.so(dht_truncate+0x172)[0x7fd39d2dd8c2] /usr/lib/glusterfs/3.3.1/xlator/performance/write-behind.so(wb_truncate+0x49e)[0x7fd39d0a769e] /usr/lib/glusterfs/3.3.1/xlator/performance/read-ahead.so(ra_truncate+0x1b2)[0x7fd39ce96302] /usr/lib/glusterfs/3.3.1/xlator/performance/io-cache.so(ioc_truncate+0x14c)[0x7fd39cc84aec] /usr/lib/libglusterfs.so.0(default_truncate+0x124)[0x7fd3a271c584] /usr/lib/glusterfs/3.3.1/xlator/performance/md-cache.so(mdc_truncate+0x13a)[0x7fd39c8625fa] /usr/lib/glusterfs/3.3.1/xlator/debug/io-stats.so(io_stats_truncate+0x14f)[0x7fd39c64bb3f] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x17b7f)[0x7fd3a0001b7f] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bd6)[0x7fd39fff0bd6] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6ae2)[0x7fd39fff0ae2] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bce)[0x7fd39fff0bce] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_continue+0x41)[0x7fd39fff0851] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_inode+0x25)[0x7fd39fff0975] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6b05)[0x7fd39fff0b05] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x6bae)[0x7fd39fff0bae] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(fuse_resolve_and_resume+0x28)[0x7fd39fff1128] /usr/lib/glusterfs/3.3.1/xlator/mount/fuse.so(+0x1af74)[0x7fd3a0004f74] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd3a20b2e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd3a1ddfccd] I tried to read all the bugs which include "Failed to get stripe-size" but they don't include anything fuse-related; anyhow it reads like https://bugzilla.redhat.com/show_bug.cgi?id=861423 which http://review.gluster.org/3904 should be solving? Thx in advance Benedikt. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users