Hello, I'm evaluating gluster for web2.0ish infrastructure provider. I'm comparing it to mogilefs and Sun's project celeste. While each has its good and bad points, gluster looks the most impressive so far. What I'm missing the most right now are: online reconfiguration and hot add/replace of bricks (storage nodes), different namespaces and/or quotas and something more than IP based authentication and authorization. I see some of those are in the roadmap already. Do you have any ETAs for future 1.x releases? What interests me the most is hot add & replace. Is mixing of different sized bricks possible? From what I can see, only unify potentialy has problems with that. What are the expirience with gluster server running on opensolaris zfs with stripe translators? Any issues with extended attributes? I'm thinking ... is there a need for data checksumming on the wire, provided that tcp already does checksuming? Right now unify and stripe translators are available ... have you thought about xor translators to implement something like raid5, raid6? Would this even make sense, if storage backend is filesystem? Both celeste and mogile have a method to specify how many redundant copies shall each object (=file) have. Have you tought about implementing something like that? In that case raid thing would not be necessary. As I understand things now, one needs to provide a separate server instance for each namespace one wants to provide. I think ZFS set a role model for how this should be done today. Are there any tehnical reasons you couldn't write a zfs like admin frontend to gluster? Also, I find it a bit suprising that basically all the redundancy config is done on the client side. All that client should get is address and auth data to mount a volume, it should have no need to care about all the "implementation details". What decisions lead you to implement this on client side? I don't think HPC world is that much different from webhosting world in this regard. Thanks for answers. -- Jure Pe?ar http://jure.pecar.org
I had a pair of servers running 1.4pre5 in AFR. they've been running fine for over a week, and suddenly today one of them had decided it just will crash anytime it tries to AFR a file. strange is, it seems to get updates form the other server. it's not up long enough to do any thorough testing, but when I do this from the "good" server: echo `hostname` `date` > /gluster/shared/file I can read the correct hostname and date from the "bad" server, but when I do the same thing on the "bad" server, it crashes instantly. running FC9 with default fuse: fuse-2.7.4-8_10.fc9.i386 I'm going to re-install fuse thinking that perhaps something got corrupted, but it's odd it happened while the servers been goign just fine for days. I turned on debugging and here's what it's producing where the log ends is where the server crashed while I was tailing the logfile: 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=21/0 active=22 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 222: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 224: CLOSE 0x8fecf58 2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] glusterfs-fuse: 225: FGETATTR 20971566 (/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) 2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] glusterfs-fuse: 225: FSTAT() /user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 223: LOOKUP() /uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such file or directory) 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=30 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(11010602), lru=20/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: 226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 224: CLOSE() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 227: FLUSH 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: 226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg => 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: 228: READ (0x8fecd50, size=4096, offset=0) 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 227: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] glusterfs-fuse: 228: READ => 3513/4096,0/3513 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 229: CLOSE 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=20/0 active=23 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 231: FLUSH 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 229: CLOSE() ERR => 0 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971566) lru=21/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] glusterfs-fuse: 230: LOOKUP() /user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971567) lru=22/0 active=21 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 231: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 232: CLOSE 0x8fecd50 2008-09-23 06:56:31 D [fuse-b here's some more from when the server rebooted +----- 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home1' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home1:storage/posix 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home1:directory:/gluster/home 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'posix-locks-home1' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so 2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] posix-locks-home1: dlsym(notify) on /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: undefined symbol: notify -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:posix-locks-home1:features/posix-locks 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:posix-locks-home1:mandatory:on 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:posix-locks-home1->home1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home2' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home2:protocol/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-type:tcp/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-host:72.36.173.218 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-timeout:10 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'server' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:server:protocol/server 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:transport-type:tcp/server 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:server->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home:read-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] fuse-options: using mount-point = /home 2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] fuse-options: using attr-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] fuse-options: using entry-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] fuse-options: using direct-io-mode = 1 2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting transport-timeout to 10 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting limits.transaction-size to 268435456 2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) 2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be done on posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP from posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: 'posix-locks-home1' supports Extended attribute 2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating new inode table with lru_limit=0 2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got GF_EVENT_PARENT_UP, attempting connect on transport 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] server: getaddrinfo failed (Name or service not known) 2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: Total data (in bytes): transfered (0), received (0) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5
I had a pair of servers running 1.4pre5 in AFR. they''ve been running fine for over a week, and suddenly today one of them had decided it just will crash anytime it tries to AFR a file. strange is, it seems to get updates form the other server. it''s not up long enough to do any thorough testing, but when I do this from the "good" server: echo `hostname` `date` > /gluster/shared/file I can read the correct hostname and date from the "bad" server, but when I do the same thing on the "bad" server, it crashes instantly. running FC9 with default fuse: fuse-2.7.4-8_10.fc9.i386 I''m going to re-install fuse thinking that perhaps something got corrupted, but it''s odd it happened while the servers been goign just fine for days. I turned on debugging and here''s what it''s producing where the log ends is where the server crashed while I was tailing the logfile: 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=21/0 active=22 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 222: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 224: CLOSE 0x8fecf58 2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] glusterfs-fuse: 225: FGETATTR 20971566 (/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) 2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] glusterfs-fuse: 225: FSTAT() /user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 223: LOOKUP() /uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such file or directory) 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=30 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(11010602), lru=20/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: 226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 224: CLOSE() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 227: FLUSH 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: 226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg => 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: 228: READ (0x8fecd50, size=4096, offset=0) 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 227: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] glusterfs-fuse: 228: READ => 3513/4096,0/3513 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 229: CLOSE 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=20/0 active=23 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 231: FLUSH 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 229: CLOSE() ERR => 0 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971566) lru=21/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] glusterfs-fuse: 230: LOOKUP() /user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971567) lru=22/0 active=21 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 231: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 232: CLOSE 0x8fecd50 2008-09-23 06:56:31 D [fuse-b here''s some more from when the server rebooted +----- 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home1:storage/posix 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home1:directory:/gluster/home 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''posix-locks-home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so 2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] posix-locks-home1: dlsym(notify) on /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: undefined symbol: notify -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:posix-locks-home1:features/posix-locks 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:posix-locks-home1:mandatory:on 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:posix-locks-home1->home1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home2:protocol/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-type:tcp/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-host:72.36.173.218 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-timeout:10 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:server:protocol/server 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:transport-type:tcp/server 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:server->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home:read-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] fuse-options: using mount-point = /home 2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] fuse-options: using attr-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] fuse-options: using entry-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] fuse-options: using direct-io-mode = 1 2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting transport-timeout to 10 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting limits.transaction-size to 268435456 2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) 2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be done on posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP from posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: ''posix-locks-home1'' supports Extended attribute 2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating new inode table with lru_limit=0 2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got GF_EVENT_PARENT_UP, attempting connect on transport 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] server: getaddrinfo failed (Name or service not known) 2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: Total data (in bytes): transfered (0), received (0) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5
I had a pair of servers running 1.4pre5 in AFR. they''ve been running fine for over a week, and suddenly today one of them had decided it just will crash anytime it tries to AFR a file. strange is, it seems to get updates form the other server. it''s not up long enough to do any thorough testing, but when I do this from the "good" server: echo `hostname` `date` > /gluster/shared/file I can read the correct hostname and date from the "bad" server, but when I do the same thing on the "bad" server, it crashes instantly. running FC9 with default fuse: fuse-2.7.4-8_10.fc9.i386 I''m going to re-install fuse thinking that perhaps something got corrupted, but it''s odd it happened while the servers been goign just fine for days. I turned on debugging and here''s what it''s producing where the log ends is where the server crashed while I was tailing the logfile: 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=21/0 active=22 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 222: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 224: CLOSE 0x8fecf58 2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] glusterfs-fuse: 225: FGETATTR 20971566 (/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) 2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] glusterfs-fuse: 225: FSTAT() /user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 223: LOOKUP() /uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such file or directory) 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=30 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(11010602), lru=20/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: 226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 224: CLOSE() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 227: FLUSH 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: 226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg => 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: 228: READ (0x8fecd50, size=4096, offset=0) 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 227: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] glusterfs-fuse: 228: READ => 3513/4096,0/3513 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 229: CLOSE 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=20/0 active=23 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 231: FLUSH 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 229: CLOSE() ERR => 0 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971566) lru=21/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] glusterfs-fuse: 230: LOOKUP() /user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971567) lru=22/0 active=21 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 231: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 232: CLOSE 0x8fecd50 2008-09-23 06:56:31 D [fuse-b here''s some more from when the server rebooted +----- 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home1:storage/posix 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home1:directory:/gluster/home 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''posix-locks-home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so 2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] posix-locks-home1: dlsym(notify) on /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: undefined symbol: notify -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:posix-locks-home1:features/posix-locks 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:posix-locks-home1:mandatory:on 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:posix-locks-home1->home1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home2:protocol/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-type:tcp/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-host:72.36.173.218 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-timeout:10 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:server:protocol/server 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:transport-type:tcp/server 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:server->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home:read-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] fuse-options: using mount-point = /home 2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] fuse-options: using attr-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] fuse-options: using entry-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] fuse-options: using direct-io-mode = 1 2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting transport-timeout to 10 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting limits.transaction-size to 268435456 2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) 2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be done on posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP from posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: ''posix-locks-home1'' supports Extended attribute 2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating new inode table with lru_limit=0 2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got GF_EVENT_PARENT_UP, attempting connect on transport 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] server: getaddrinfo failed (Name or service not known) 2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: Total data (in bytes): transfered (0), received (0) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5
I had a pair of servers running 1.4pre5 in AFR. they''ve been running fine for over a week, and suddenly today one of them had decided it just will crash anytime it tries to AFR a file. strange is, it seems to get updates form the other server. it''s not up long enough to do any thorough testing, but when I do this from the "good" server: echo `hostname` `date` > /gluster/shared/file I can read the correct hostname and date from the "bad" server, but when I do the same thing on the "bad" server, it crashes instantly. running FC9 with default fuse: fuse-2.7.4-8_10.fc9.i386 I''m going to re-install fuse thinking that perhaps something got corrupted, but it''s odd it happened while the servers been goign just fine for days. I turned on debugging and here''s what it''s producing where the log ends is where the server crashed while I was tailing the logfile: 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=21/0 active=22 purge=29 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 222: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 224: CLOSE 0x8fecf58 2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] glusterfs-fuse: 225: FGETATTR 20971566 (/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) 2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] glusterfs-fuse: 225: FSTAT() /user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] glusterfs-fuse: 223: LOOKUP() /uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such file or directory) 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: retiring inode(0) lru=21/0 active=21 purge=30 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(11010602), lru=20/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: 226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 224: CLOSE() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 227: FLUSH 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: 226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg => 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: 228: READ (0x8fecd50, size=4096, offset=0) 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 227: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] glusterfs-fuse: 228: READ => 3513/4096,0/3513 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 229: CLOSE 0x8fece28 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: 230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: activating inode(0), lru=20/0 active=23 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: 231: FLUSH 0x8fecd50 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 229: CLOSE() ERR => 0 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971566) lru=21/0 active=22 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] glusterfs-fuse: 230: LOOKUP() /user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: passivating inode(20971567) lru=22/0 active=21 purge=30 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] glusterfs-fuse: 231: FLUSH() ERR => 0 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] glusterfs-fuse: 232: CLOSE 0x8fecd50 2008-09-23 06:56:31 D [fuse-b here''s some more from when the server rebooted +----- 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home1:storage/posix 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home1:directory:/gluster/home 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''posix-locks-home1'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so 2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] posix-locks-home1: dlsym(notify) on /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: undefined symbol: notify -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:posix-locks-home1:features/posix-locks 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:posix-locks-home1:mandatory:on 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:posix-locks-home1->home1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home2:protocol/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-type:tcp/client 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-host:72.36.173.218 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:remote-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home2:transport-timeout:10 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:server:protocol/server 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:transport-type:tcp/server 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:server->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: Option:home:read-subvolume:posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->posix-locks-home1 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict option validation is not enforced -- neglecting 2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] fuse-options: using mount-point = /home 2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] fuse-options: using attr-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] fuse-options: using entry-timeout = 1 2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] fuse-options: using direct-io-mode = 1 2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting transport-timeout to 10 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting limits.transaction-size to 268435456 2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) 2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be done on posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP from posix-locks-home1 2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: ''posix-locks-home1'' supports Extended attribute 2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating new inode table with lru_limit=0 2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) 2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got GF_EVENT_PARENT_UP, attempting connect on transport 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so 2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] server: getaddrinfo failed (Name or service not known) 2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: Total data (in bytes): transfered (0), received (0) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5
Hi Keith, I've attached a patch containing the fix. Apply the patch, rebuild and restart glusterfs. regards, On Tue, Sep 23, 2008 at 6:09 PM, Keith Freedman <freedman at freeformit.com>wrote:> I had a pair of servers running 1.4pre5 in AFR. > they've been running fine for over a week, and suddenly today one of > them had decided it just will crash anytime it tries to AFR a file. > > strange is, it seems to get updates form the other server. > it's not up long enough to do any thorough testing, but when I do > this from the "good" server: > echo `hostname` `date` > /gluster/shared/file > I can read the correct hostname and date from the "bad" server, but > when I do the same thing on the "bad" server, it crashes instantly. > > running FC9 with default fuse: > fuse-2.7.4-8_10.fc9.i386 > > I'm going to re-install fuse thinking that perhaps something got > corrupted, but it's odd it happened while the servers been goign just > fine for days. > > I turned on debugging and here's what it's producing > where the log ends is where the server crashed while I was tailing the > logfile: > 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > retiring inode(0) lru=21/0 active=21 purge=29 > 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > 223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess > 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > activating inode(0), lru=21/0 active=22 purge=29 > 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > glusterfs-fuse: 222: FLUSH() ERR => 0 > 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > glusterfs-fuse: 224: CLOSE 0x8fecf58 > 2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] > glusterfs-fuse: 225: FGETATTR 20971566 > (/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) > 2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] > glusterfs-fuse: 225: FSTAT() > /user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 > 2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] > glusterfs-fuse: 223: LOOKUP() > /uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such > file or directory) > 2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > retiring inode(0) lru=21/0 active=21 purge=30 > 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > activating inode(11010602), lru=20/0 active=22 purge=30 > 2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: > 226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > glusterfs-fuse: 224: CLOSE() ERR => 0 > 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > 227: FLUSH 0x8fece28 > 2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: > 226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg => > 0x8fecd50 > 2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: > 228: READ (0x8fecd50, size=4096, offset=0) > 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > glusterfs-fuse: 227: FLUSH() ERR => 0 > 2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] > glusterfs-fuse: 228: READ => 3513/4096,0/3513 > 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > glusterfs-fuse: 229: CLOSE 0x8fece28 > 2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > 230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php > 2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > 2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > activating inode(0), lru=20/0 active=23 purge=30 > 2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > 231: FLUSH 0x8fecd50 > 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > glusterfs-fuse: 229: CLOSE() ERR => 0 > 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > passivating inode(20971566) lru=21/0 active=22 purge=30 > 2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] > glusterfs-fuse: 230: LOOKUP() > /user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 > 2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > passivating inode(20971567) lru=22/0 active=21 purge=30 > 2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > glusterfs-fuse: 231: FLUSH() ERR => 0 > 2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > glusterfs-fuse: 232: CLOSE 0x8fecd50 > 2008-09-23 06:56:31 D [fuse-b > > here's some more from when the server rebooted > +----- > 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home1' > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so > 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home1:storage/posix > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home1:directory:/gluster/home > 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 > 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > 'posix-locks-home1' > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so > 2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] > posix-locks-home1: dlsym(notify) on > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: > undefined symbol: notify -- neglecting > 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:posix-locks-home1:features/posix-locks > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:posix-locks-home1:mandatory:on > 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > child:posix-locks-home1->home1 > 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: > end:posix-locks-home1 > 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home2' > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so > 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home2:protocol/client > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home2:transport-type:tcp/client > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home2:remote-host:72.36.173.218 > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home2:remote-subvolume:posix-locks-home1 > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home2:transport-timeout:10 > 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 > 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > 'server' > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so > 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:server:protocol/server > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:server:transport-type:tcp/server > 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > child:server->posix-locks-home1 > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 > 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server > 2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home' > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so > 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict > option validation is not enforced -- neglecting > 2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home:cluster/afr > 2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > Option:home:read-subvolume:posix-locks-home1 > 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > child:home->posix-locks-home1 > 2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 > 2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home > 2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so > 2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict > option validation is not enforced -- neglecting > 2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 > 2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] > fuse-options: using mount-point = /home > 2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] > fuse-options: using attr-timeout = 1 > 2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] > fuse-options: using entry-timeout = 1 > 2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] > fuse-options: using direct-io-mode = 1 > 2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting > transport-timeout to 10 > 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > 2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting > limits.transaction-size to 268435456 > 2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled > (default) > 2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be > done on posix-locks-home1 > 2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP > from posix-locks-home1 > 2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: > 'posix-locks-home1' supports Extended attribute > 2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating > new inode table with lru_limit=0 > 2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > 2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got > GF_EVENT_PARENT_UP, attempting connect on transport > 2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > 2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] > server: getaddrinfo failed (Name or service not known) > 2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: > Total data (in bytes): transfered (0), received (0) > pending frames: > > Signal received: 11 > configuration details:argp 1 > backtrace 1 > dlfcn 1 > fdatasync 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > tv_nsec 1 > package-string: glusterfs 1.4.0pre5 > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >-- Raghavendra G A centipede was happy quite, until a toad in fun, Said, "Prey, which leg comes after which?", This raised his doubts to such a pitch, He fell flat into the ditch, Not knowing how to run. -Anonymous -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080923/2e87f010/attachment.html> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: name-segfault.diff.txt URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080923/2e87f010/attachment.txt>
as a followup.. I have shutdown the "broken" one in the pair since it kept crashing. the working one is running on it''s own but gluster dies every 10 mins or so. seems 1.4pre5 doesn''t like being an AFR client all on it''s own? I''m going to see if it works with only itself as the AFR subvolumes list 2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: (path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5 /lib64/libc.so.6[0x300d0322a0] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] /usr/local/lib/libglusterfs.so.0[0x125c5b] /usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] /usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] /lib64/libfuse.so.2[0x10824b2] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] /lib64/libpthread.so.0[0x300dc0729a] /lib64/libc.so.6(clone+0x6d)[0x300d0e439d] --------- At 07:09 AM 9/23/2008, Keith Freedman wrote:>I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
as a followup.. I have shutdown the "broken" one in the pair since it kept crashing. the working one is running on it''s own but gluster dies every 10 mins or so. seems 1.4pre5 doesn''t like being an AFR client all on it''s own? I''m going to see if it works with only itself as the AFR subvolumes list 2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: (path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5 /lib64/libc.so.6[0x300d0322a0] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] /usr/local/lib/libglusterfs.so.0[0x125c5b] /usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] /usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] /lib64/libfuse.so.2[0x10824b2] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] /lib64/libpthread.so.0[0x300dc0729a] /lib64/libc.so.6(clone+0x6d)[0x300d0e439d] --------- At 07:09 AM 9/23/2008, Keith Freedman wrote:>I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
as a followup.. I have shutdown the "broken" one in the pair since it kept crashing. the working one is running on it''s own but gluster dies every 10 mins or so. seems 1.4pre5 doesn''t like being an AFR client all on it''s own? I''m going to see if it works with only itself as the AFR subvolumes list 2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) 2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: (path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) op_ret=-1 op_errno=107(Transport endpoint is not connected) pending frames: Signal received: 11 configuration details:argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 tv_nsec 1 package-string: glusterfs 1.4.0pre5 /lib64/libc.so.6[0x300d0322a0] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] /usr/local/lib/libglusterfs.so.0[0x125c5b] /usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] /usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] /lib64/libfuse.so.2[0x10824b2] /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] /lib64/libpthread.so.0[0x300dc0729a] /lib64/libc.so.6(clone+0x6d)[0x300d0e439d] --------- At 07:09 AM 9/23/2008, Keith Freedman wrote:>I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
ok.. I''ll do that to both servers. Keith At 07:32 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, >I''ve attached a patch containing the fix. Apply the patch, rebuild >and restart glusterfs. > >regards, > >On Tue, Sep 23, 2008 at 6:09 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.218>72.36.173.218,<http://127.0.0.1>127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous >Content-Type: text/plain; name=name-segfault.diff.txt >X-Attachment-Id: f_flgpu5t80 >Content-Disposition: attachment; filename=name-segfault.diff.txt >
ok.. I''ll do that to both servers. Keith At 07:32 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, >I''ve attached a patch containing the fix. Apply the patch, rebuild >and restart glusterfs. > >regards, > >On Tue, Sep 23, 2008 at 6:09 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.218>72.36.173.218,<http://127.0.0.1>127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous >Content-Type: text/plain; name=name-segfault.diff.txt >X-Attachment-Id: f_flgpu5t80 >Content-Disposition: attachment; filename=name-segfault.diff.txt >
ok.. I''ll do that to both servers. Keith At 07:32 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, >I''ve attached a patch containing the fix. Apply the patch, rebuild >and restart glusterfs. > >regards, > >On Tue, Sep 23, 2008 at 6:09 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >I had a pair of servers running 1.4pre5 in AFR. >they''ve been running fine for over a week, and suddenly today one of >them had decided it just will crash anytime it tries to AFR a file. > >strange is, it seems to get updates form the other server. >it''s not up long enough to do any thorough testing, but when I do >this from the "good" server: >echo `hostname` `date` > /gluster/shared/file >I can read the correct hostname and date from the "bad" server, but >when I do the same thing on the "bad" server, it crashes instantly. > >running FC9 with default fuse: >fuse-2.7.4-8_10.fc9.i386 > >I''m going to re-install fuse thinking that perhaps something got >corrupted, but it''s odd it happened while the servers been goign just >fine for days. > >I turned on debugging and here''s what it''s producing >where the log ends is where the server crashed while I was tailing >the logfile: >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=21/0 active=22 purge=29 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 222: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 224: CLOSE 0x8fecf58 >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] >glusterfs-fuse: 225: FGETATTR 20971566 >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] >glusterfs-fuse: 225: FSTAT() >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] >glusterfs-fuse: 223: LOOKUP() >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such >file or directory) >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: >retiring inode(0) lru=21/0 active=21 purge=30 >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(11010602), lru=20/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 224: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >227: FLUSH 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg >=> 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: >228: READ (0x8fecd50, size=4096, offset=0) >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 227: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] >glusterfs-fuse: 228: READ => 3513/4096,0/3513 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 229: CLOSE 0x8fece28 >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: >activating inode(0), lru=20/0 active=23 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: >231: FLUSH 0x8fecd50 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 229: CLOSE() ERR => 0 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971566) lru=21/0 active=22 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] >glusterfs-fuse: 230: LOOKUP() >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: >passivating inode(20971567) lru=22/0 active=21 purge=30 >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] >glusterfs-fuse: 231: FLUSH() ERR => 0 >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] >glusterfs-fuse: 232: CLOSE 0x8fecd50 >2008-09-23 06:56:31 D [fuse-b > >here''s some more from when the server rebooted >+----- >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home1:storage/posix >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home1:directory:/gluster/home >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for >''posix-locks-home1'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] >posix-locks-home1: dlsym(notify) on >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: >undefined symbol: notify -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:posix-locks-home1:features/posix-locks >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:posix-locks-home1:mandatory:on >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:posix-locks-home1->home1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:home2:protocol/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-type:tcp/client >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:remote-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home2:transport-timeout:10 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: >Type:server:protocol/server >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:transport-type:tcp/server >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:server->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.218>72.36.173.218,<http://127.0.0.1>127.0.0.1 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: >Option:home:read-subvolume:posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: >child:home->posix-locks-home1 >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict >option validation is not enforced -- neglecting >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] >fuse-options: using mount-point = /home >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] >fuse-options: using attr-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] >fuse-options: using entry-timeout = 1 >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] >fuse-options: using direct-io-mode = 1 >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting >transport-timeout to 10 >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting >limits.transaction-size to 268435456 >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be >done on posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP >from posix-locks-home1 >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: >''posix-locks-home1'' supports Extended attribute >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating >new inode table with lru_limit=0 >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0) >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got >GF_EVENT_PARENT_UP, attempting connect on transport >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] >server: getaddrinfo failed (Name or service not known) >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: >Total data (in bytes): transfered (0), received (0) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 > > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous >Content-Type: text/plain; name=name-segfault.diff.txt >X-Attachment-Id: f_flgpu5t80 >Content-Disposition: attachment; filename=name-segfault.diff.txt >
Hi Keith, Work is being done on AFR to make it stable. Please wait for a stable release. On Tue, Sep 23, 2008 at 6:41 PM, Keith Freedman <freedman at freeformit.com>wrote:> as a followup.. I have shutdown the "broken" one in the pair since > it kept crashing. > the working one is running on it's own but gluster dies every 10 mins or > so. > seems 1.4pre5 doesn't like being an AFR client all on it's own? > > I'm going to see if it works with only itself as the AFR subvolumes list > > > > > 2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) > op_ret=-1 op_errno=107(Transport endpoint is not connected) > 2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) > op_ret=-1 op_errno=107(Transport endpoint is not connected) > 2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: > (path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) > op_ret=-1 op_errno=107(Transport endpoint is not connected) > pending frames: > > Signal received: 11 > configuration details:argp 1 > backtrace 1 > dlfcn 1 > fdatasync 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > tv_nsec 1 > package-string: glusterfs 1.4.0pre5 > /lib64/libc.so.6[0x300d0322a0] > > /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] > > /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] > > /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] > /usr/local/lib/libglusterfs.so.0[0x125c5b] > /usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] > /usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] > > /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] > > /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] > /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] > /lib64/libfuse.so.2[0x10824b2] > /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] > /lib64/libpthread.so.0[0x300dc0729a] > /lib64/libc.so.6(clone+0x6d)[0x300d0e439d] > --------- > > > At 07:09 AM 9/23/2008, Keith Freedman wrote: > >I had a pair of servers running 1.4pre5 in AFR. > >they've been running fine for over a week, and suddenly today one of > >them had decided it just will crash anytime it tries to AFR a file. > > > >strange is, it seems to get updates form the other server. > >it's not up long enough to do any thorough testing, but when I do > >this from the "good" server: > >echo `hostname` `date` > /gluster/shared/file > >I can read the correct hostname and date from the "bad" server, but > >when I do the same thing on the "bad" server, it crashes instantly. > > > >running FC9 with default fuse: > >fuse-2.7.4-8_10.fc9.i386 > > > >I'm going to re-install fuse thinking that perhaps something got > >corrupted, but it's odd it happened while the servers been goign just > >fine for days. > > > >I turned on debugging and here's what it's producing > >where the log ends is where the server crashed while I was tailing > >the logfile: > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=21/0 active=22 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 222: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 224: CLOSE 0x8fecf58 > >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] > >glusterfs-fuse: 225: FGETATTR 20971566 > > >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) > >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] > >glusterfs-fuse: 225: FSTAT() > >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => > 20971566 > >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] > >glusterfs-fuse: 223: LOOKUP() > >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such > >file or directory) > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=30 > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(11010602), lru=20/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: > >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 224: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >227: FLUSH 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: > >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >=> 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: > >228: READ (0x8fecd50, size=4096, offset=0) > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 227: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] > >glusterfs-fuse: 228: READ => 3513/4096,0/3513 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 229: CLOSE 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >230: LOOKUP > /user2/public_html/shopping/var/run/classes/kernel/Database.php > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=20/0 active=23 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >231: FLUSH 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 229: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971566) lru=21/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] > >glusterfs-fuse: 230: LOOKUP() > >/user2/public_html/shopping/var/run/classes/kernel/Database.php => > 20971567 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971567) lru=22/0 active=21 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 231: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 232: CLOSE 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-b > > > >here's some more from when the server rebooted > >+----- > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > 'home1' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home1:storage/posix > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home1:directory:/gluster/home > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > >'posix-locks-home1' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so > >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] > >posix-locks-home1: dlsym(notify) on > >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: > >undefined symbol: notify -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:posix-locks-home1:features/posix-locks > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:posix-locks-home1:mandatory:on > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:posix-locks-home1->home1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: > end:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > 'home2' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home2:protocol/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-type:tcp/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-host:72.36.173.218 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-timeout:10 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > 'server' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:server:protocol/server > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:transport-type:tcp/server > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:server->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:auth.addr.posix-locks-home1.allow:72.36.173.218,127.0.0.1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home:cluster/afr > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home:read-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:home->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid > 1145 > >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] > >fuse-options: using mount-point = /home > >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] > >fuse-options: using attr-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] > >fuse-options: using entry-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] > >fuse-options: using direct-io-mode = 1 > >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting > >transport-timeout to 10 > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file > /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting > >limits.transaction-size to 268435456 > >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled > (default) > >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be > >done on posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP > >from posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: > >'posix-locks-home1' supports Extended attribute > >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating > >new inode table with lru_limit=0 > >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create > inode(0) > >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got > >GF_EVENT_PARENT_UP, attempting connect on transport > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file > /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] > >server: getaddrinfo failed (Name or service not known) > >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: > >Total data (in bytes): transfered (0), received (0) > >pending frames: > > > >Signal received: 11 > >configuration details:argp 1 > >backtrace 1 > >dlfcn 1 > >fdatasync 1 > >libpthread 1 > >llistxattr 1 > >setfsid 1 > >spinlock 1 > >epoll.h 1 > >xattr.h 1 > >tv_nsec 1 > >package-string: glusterfs 1.4.0pre5 > > > > > > > >_______________________________________________ > >Gluster-users mailing list > >Gluster-users at gluster.org > >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >-- Raghavendra G A centipede was happy quite, until a toad in fun, Said, "Prey, which leg comes after which?", This raised his doubts to such a pitch, He fell flat into the ditch, Not knowing how to run. -Anonymous -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080923/f931faa6/attachment.html>
ok, but what do I until then? and it''s so odd it just suddenly broke after all this time. At 07:50 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, > >Work is being done on AFR to make it stable. Please wait for a stable release. > >On Tue, Sep 23, 2008 at 6:41 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >as a followup.. I have shutdown the "broken" one in the pair since >it kept crashing. >the working one is running on it''s own but gluster dies every 10 mins or so. >seems 1.4pre5 doesn''t like being an AFR client all on it''s own? > >I''m going to see if it works with only itself as the AFR subvolumes list > > > > >2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: >(path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 >/lib64/libc.so.6[0x300d0322a0] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] >/usr/local/lib/libglusterfs.so.0[0x125c5b] >/usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] >/usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] >/lib64/libfuse.so.2[0x10824b2] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] >/lib64/libpthread.so.0[0x300dc0729a] >/lib64/libc.so.6(clone+0x6d)[0x300d0e439d] >--------- > > >At 07:09 AM 9/23/2008, Keith Freedman wrote: > >I had a pair of servers running 1.4pre5 in AFR. > >they''ve been running fine for over a week, and suddenly today one of > >them had decided it just will crash anytime it tries to AFR a file. > > > >strange is, it seems to get updates form the other server. > >it''s not up long enough to do any thorough testing, but when I do > >this from the "good" server: > >echo `hostname` `date` > /gluster/shared/file > >I can read the correct hostname and date from the "bad" server, but > >when I do the same thing on the "bad" server, it crashes instantly. > > > >running FC9 with default fuse: > >fuse-2.7.4-8_10.fc9.i386 > > > >I''m going to re-install fuse thinking that perhaps something got > >corrupted, but it''s odd it happened while the servers been goign just > >fine for days. > > > >I turned on debugging and here''s what it''s producing > >where the log ends is where the server crashed while I was tailing > >the logfile: > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=21/0 active=22 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 222: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 224: CLOSE 0x8fecf58 > >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] > >glusterfs-fuse: 225: FGETATTR 20971566 > >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) > >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] > >glusterfs-fuse: 225: FSTAT() > >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 > >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] > >glusterfs-fuse: 223: LOOKUP() > >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such > >file or directory) > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=30 > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(11010602), lru=20/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: > >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 224: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >227: FLUSH 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: > >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >=> 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: > >228: READ (0x8fecd50, size=4096, offset=0) > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 227: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] > >glusterfs-fuse: 228: READ => 3513/4096,0/3513 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 229: CLOSE 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=20/0 active=23 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >231: FLUSH 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 229: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971566) lru=21/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] > >glusterfs-fuse: 230: LOOKUP() > >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971567) lru=22/0 active=21 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 231: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 232: CLOSE 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-b > > > >here''s some more from when the server rebooted > >+----- > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home1:storage/posix > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home1:directory:/gluster/home > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > >''posix-locks-home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so > >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] > >posix-locks-home1: dlsym(notify) on > >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: > >undefined symbol: notify -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:posix-locks-home1:features/posix-locks > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:posix-locks-home1:mandatory:on > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:posix-locks-home1->home1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home2:protocol/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-type:tcp/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-timeout:10 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:server:protocol/server > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:transport-type:tcp/server > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:server->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.2 > 18>72.36.173.218,<http://127.0.0.1>127.0.0.1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home:cluster/afr > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home:read-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:home->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 > >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] > >fuse-options: using mount-point = /home > >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] > >fuse-options: using attr-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] > >fuse-options: using entry-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] > >fuse-options: using direct-io-mode = 1 > >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting > >transport-timeout to 10 > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting > >limits.transaction-size to 268435456 > >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) > >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be > >done on posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP > >from posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: > >''posix-locks-home1'' supports Extended attribute > >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating > >new inode table with lru_limit=0 > >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got > >GF_EVENT_PARENT_UP, attempting connect on transport > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] > >server: getaddrinfo failed (Name or service not known) > >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: > >Total data (in bytes): transfered (0), received (0) > >pending frames: > > > >Signal received: 11 > >configuration details:argp 1 > >backtrace 1 > >dlfcn 1 > >fdatasync 1 > >libpthread 1 > >llistxattr 1 > >setfsid 1 > >spinlock 1 > >epoll.h 1 > >xattr.h 1 > >tv_nsec 1 > >package-string: glusterfs 1.4.0pre5 > > > > > > > >_______________________________________________ > >Gluster-users mailing list > ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org > >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous
ok, but what do I until then? and it''s so odd it just suddenly broke after all this time. At 07:50 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, > >Work is being done on AFR to make it stable. Please wait for a stable release. > >On Tue, Sep 23, 2008 at 6:41 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >as a followup.. I have shutdown the "broken" one in the pair since >it kept crashing. >the working one is running on it''s own but gluster dies every 10 mins or so. >seems 1.4pre5 doesn''t like being an AFR client all on it''s own? > >I''m going to see if it works with only itself as the AFR subvolumes list > > > > >2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: >(path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 >/lib64/libc.so.6[0x300d0322a0] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] >/usr/local/lib/libglusterfs.so.0[0x125c5b] >/usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] >/usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] >/lib64/libfuse.so.2[0x10824b2] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] >/lib64/libpthread.so.0[0x300dc0729a] >/lib64/libc.so.6(clone+0x6d)[0x300d0e439d] >--------- > > >At 07:09 AM 9/23/2008, Keith Freedman wrote: > >I had a pair of servers running 1.4pre5 in AFR. > >they''ve been running fine for over a week, and suddenly today one of > >them had decided it just will crash anytime it tries to AFR a file. > > > >strange is, it seems to get updates form the other server. > >it''s not up long enough to do any thorough testing, but when I do > >this from the "good" server: > >echo `hostname` `date` > /gluster/shared/file > >I can read the correct hostname and date from the "bad" server, but > >when I do the same thing on the "bad" server, it crashes instantly. > > > >running FC9 with default fuse: > >fuse-2.7.4-8_10.fc9.i386 > > > >I''m going to re-install fuse thinking that perhaps something got > >corrupted, but it''s odd it happened while the servers been goign just > >fine for days. > > > >I turned on debugging and here''s what it''s producing > >where the log ends is where the server crashed while I was tailing > >the logfile: > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=21/0 active=22 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 222: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 224: CLOSE 0x8fecf58 > >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] > >glusterfs-fuse: 225: FGETATTR 20971566 > >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) > >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] > >glusterfs-fuse: 225: FSTAT() > >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 > >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] > >glusterfs-fuse: 223: LOOKUP() > >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such > >file or directory) > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=30 > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(11010602), lru=20/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: > >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 224: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >227: FLUSH 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: > >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >=> 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: > >228: READ (0x8fecd50, size=4096, offset=0) > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 227: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] > >glusterfs-fuse: 228: READ => 3513/4096,0/3513 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 229: CLOSE 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=20/0 active=23 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >231: FLUSH 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 229: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971566) lru=21/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] > >glusterfs-fuse: 230: LOOKUP() > >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971567) lru=22/0 active=21 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 231: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 232: CLOSE 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-b > > > >here''s some more from when the server rebooted > >+----- > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home1:storage/posix > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home1:directory:/gluster/home > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > >''posix-locks-home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so > >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] > >posix-locks-home1: dlsym(notify) on > >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: > >undefined symbol: notify -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:posix-locks-home1:features/posix-locks > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:posix-locks-home1:mandatory:on > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:posix-locks-home1->home1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home2:protocol/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-type:tcp/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-timeout:10 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:server:protocol/server > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:transport-type:tcp/server > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:server->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.2 > 18>72.36.173.218,<http://127.0.0.1>127.0.0.1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home:cluster/afr > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home:read-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:home->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 > >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] > >fuse-options: using mount-point = /home > >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] > >fuse-options: using attr-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] > >fuse-options: using entry-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] > >fuse-options: using direct-io-mode = 1 > >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting > >transport-timeout to 10 > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting > >limits.transaction-size to 268435456 > >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) > >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be > >done on posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP > >from posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: > >''posix-locks-home1'' supports Extended attribute > >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating > >new inode table with lru_limit=0 > >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got > >GF_EVENT_PARENT_UP, attempting connect on transport > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] > >server: getaddrinfo failed (Name or service not known) > >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: > >Total data (in bytes): transfered (0), received (0) > >pending frames: > > > >Signal received: 11 > >configuration details:argp 1 > >backtrace 1 > >dlfcn 1 > >fdatasync 1 > >libpthread 1 > >llistxattr 1 > >setfsid 1 > >spinlock 1 > >epoll.h 1 > >xattr.h 1 > >tv_nsec 1 > >package-string: glusterfs 1.4.0pre5 > > > > > > > >_______________________________________________ > >Gluster-users mailing list > ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org > >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous
ok, but what do I until then? and it''s so odd it just suddenly broke after all this time. At 07:50 AM 9/23/2008, Raghavendra G wrote:>Hi Keith, > >Work is being done on AFR to make it stable. Please wait for a stable release. > >On Tue, Sep 23, 2008 at 6:41 PM, Keith Freedman ><<mailto:freedman at freeformit.com>freedman at freeformit.com> wrote: >as a followup.. I have shutdown the "broken" one in the pair since >it kept crashing. >the working one is running on it''s own but gluster dies every 10 mins or so. >seems 1.4pre5 doesn''t like being an AFR client all on it''s own? > >I''m going to see if it works with only itself as the AFR subvolumes list > > > > >2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home: >(path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2) >op_ret=-1 op_errno=107(Transport endpoint is not connected) >pending frames: > >Signal received: 11 >configuration details:argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >tv_nsec 1 >package-string: glusterfs 1.4.0pre5 >/lib64/libc.so.6[0x300d0322a0] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75] >/usr/local/lib/libglusterfs.so.0[0x125c5b] >/usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa] >/usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8] >/lib64/libfuse.so.2[0x10824b2] >/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35] >/lib64/libpthread.so.0[0x300dc0729a] >/lib64/libc.so.6(clone+0x6d)[0x300d0e439d] >--------- > > >At 07:09 AM 9/23/2008, Keith Freedman wrote: > >I had a pair of servers running 1.4pre5 in AFR. > >they''ve been running fine for over a week, and suddenly today one of > >them had decided it just will crash anytime it tries to AFR a file. > > > >strange is, it seems to get updates form the other server. > >it''s not up long enough to do any thorough testing, but when I do > >this from the "good" server: > >echo `hostname` `date` > /gluster/shared/file > >I can read the correct hostname and date from the "bad" server, but > >when I do the same thing on the "bad" server, it crashes instantly. > > > >running FC9 with default fuse: > >fuse-2.7.4-8_10.fc9.i386 > > > >I''m going to re-install fuse thinking that perhaps something got > >corrupted, but it''s odd it happened while the servers been goign just > >fine for days. > > > >I turned on debugging and here''s what it''s producing > >where the log ends is where the server crashed while I was tailing > >the logfile: > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=21/0 active=22 purge=29 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 222: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 224: CLOSE 0x8fecf58 > >2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr] > >glusterfs-fuse: 225: FGETATTR 20971566 > >(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28) > >2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk] > >glusterfs-fuse: 225: FSTAT() > >/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566 > >2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk] > >glusterfs-fuse: 223: LOOKUP() > >/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such > >file or directory) > >2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode: > >retiring inode(0) lru=21/0 active=21 purge=30 > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(11010602), lru=20/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse: > >226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 224: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >227: FLUSH 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse: > >226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg > >=> 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse: > >228: READ (0x8fecd50, size=4096, offset=0) > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 227: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk] > >glusterfs-fuse: 228: READ => 3513/4096,0/3513 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 229: CLOSE 0x8fece28 > >2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse: > >230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php > >2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode: > >activating inode(0), lru=20/0 active=23 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse: > >231: FLUSH 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 229: CLOSE() ERR => 0 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971566) lru=21/0 active=22 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk] > >glusterfs-fuse: 230: LOOKUP() > >/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567 > >2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode: > >passivating inode(20971567) lru=22/0 active=21 purge=30 > >2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk] > >glusterfs-fuse: 231: FLUSH() ERR => 0 > >2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release] > >glusterfs-fuse: 232: CLOSE 0x8fecd50 > >2008-09-23 06:56:31 D [fuse-b > > > >here''s some more from when the server rebooted > >+----- > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home1:storage/posix > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home1:directory:/gluster/home > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for > >''posix-locks-home1'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file > /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so > >2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type] > >posix-locks-home1: dlsym(notify) on > >/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so: > >undefined symbol: notify -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:posix-locks-home1:features/posix-locks > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:posix-locks-home1:mandatory:on > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:posix-locks-home1->home1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home2'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:home2:protocol/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-type:tcp/client > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-host:<http://72.36.173.218>72.36.173.218 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:remote-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home2:transport-timeout:10 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2 > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''server'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > >Type:server:protocol/server > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:transport-type:tcp/server > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:server->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:server:auth.addr.posix-locks-home1.allow:<http://72.36.173.2 > 18>72.36.173.218,<http://127.0.0.1>127.0.0.1 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server > >2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for ''home'' > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [spec.y:219:section_type] parser: > Type:home:cluster/afr > >2008-09-23 07:04:39 D [spec.y:285:section_option] parser: > >Option:home:read-subvolume:posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: > >child:home->posix-locks-home1 > >2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2 > >2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home > >2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt > >to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so > >2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict > >option validation is not enforced -- neglecting > >2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145 > >2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate] > >fuse-options: using mount-point = /home > >2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate] > >fuse-options: using attr-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate] > >fuse-options: using entry-timeout = 1 > >2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate] > >fuse-options: using direct-io-mode = 1 > >2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting > >transport-timeout to 10 > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting > >limits.transaction-size to 268435456 > >2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default) > >2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be > >done on posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP > >from posix-locks-home1 > >2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home: > >''posix-locks-home1'' supports Extended attribute > >2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating > >new inode table with lru_limit=0 > >2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: > create inode(0) > >2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got > >GF_EVENT_PARENT_UP, attempting connect on transport > >2008-09-23 07:04:39 D [transport.c:104:transport_load] transport: > >attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so > >2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr] > >server: getaddrinfo failed (Name or service not known) > >2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs: > >Total data (in bytes): transfered (0), received (0) > >pending frames: > > > >Signal received: 11 > >configuration details:argp 1 > >backtrace 1 > >dlfcn 1 > >fdatasync 1 > >libpthread 1 > >llistxattr 1 > >setfsid 1 > >spinlock 1 > >epoll.h 1 > >xattr.h 1 > >tv_nsec 1 > >package-string: glusterfs 1.4.0pre5 > > > > > > > >_______________________________________________ > >Gluster-users mailing list > ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org > >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > >_______________________________________________ >Gluster-users mailing list ><mailto:Gluster-users at gluster.org>Gluster-users at gluster.org >http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > >-- >Raghavendra G > >A centipede was happy quite, until a toad in fun, >Said, "Prey, which leg comes after which?", >This raised his doubts to such a pitch, >He fell flat into the ditch, >Not knowing how to run. >-Anonymous
Hi all,
we are running stress test and virtual machine (VMWARE) on expensive
hardware, after trying glusterfs on cheaper hardware.
Our platform is as follow:
- 2 node (SunFire X4150),
- OS: Debian Lenny , kernel 2.6.24, glusterfs 1.3.12, fuse-2.7.3glfs10
(from your repository)
Glusterfs and fuse are compiled on both nodes; setup provides a
server/client configuration with afr and 2 bricks per node
We have 2 NIC per node directly connected for improved reliability.
the config file is reported below.
>From log files of both nodes we have a lot of next messagges:
"
...
2008-09-26 06:25:21 E [unify.c:182:unify_lookup_cbk] unify: afr2
returned 2
2008-09-26 06:25:21 E [unify.c:265:unify_lookup_cbk] unify: Revalidate
failed for /virtual_machines/vm-debug-Debian/vmware-0.log
2008-09-26 06:25:21 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse:
13393848: (34) /virtual_machines/vm-debug-Debian/vmware-0.log => -1 (2)
2008-09-26 06:25:21 W [fuse-bridge.c:411:fuse_entry_cbk] glusterfs-fuse:
13393849: (34) /virtual_machines/vm-debug-Debian/vmware.log => 101514846
Rehashing because st_nlink less than dentry maps
...
"
The alert refers to many files.
I would like to ask you what is the meaning of this log.
Any help is really appreciated.
Kind regards,
Davide
------------------------------ Config file
--------------------------------
# dataspace on storage1
volume ds1
type storage/posix
option directory /export/glusterfs1
end-volume
# dataspace on storage2
volume ds2
type storage/posix
option directory /export/glusterfs2
end-volume
# namespace for unify
volume ns1
type storage/posix
option directory /var/glusterfs/ns
end-volume
# posix locks
volume ds-locks1
type features/posix-locks
subvolumes ds1
end-volume
# posix locks
volume ds-locks2
type features/posix-locks
subvolumes ds2
end-volume
volume ds-threads1
type performance/io-threads
option thread-count 4
option cache-size 128MB
subvolumes ds-locks1
end-volume
volume ds-threads2
type performance/io-threads
option thread-count 4
option cache-size 128MB
subvolumes ds-locks2
end-volume
volume server
type protocol/server
option transport-type tcp/server
subvolumes ds-threads1 ds-threads2 ns1
# storage network access only
option auth.ip.ds-threads1.allow *
option auth.ip.ds-threads2.allow *
option auth.ip.ns1.allow *
# option auth.ip.gfs.allow *
end-volume
# dataspace 1 on the other node
volume ds3
type protocol/client
option transport-type tcp/client
option remote-host 192.168.1.2 # storage network
option remote-subvolume ds-threads1
# option transport-timeout 10 # value in seconds; it should
be set relatively low
end-volume
# dataspace 2 on the other node
volume ds4
type protocol/client
option transport-type tcp/client
option remote-host 192.168.1.2 # storage network
option remote-subvolume ds-threads2
# option transport-timeout 10 # value in seconds; it should
be set relatively low
end-volume
# namespace 2 on the other node
volume ns2
type protocol/client
option transport-type tcp/client
option remote-host 192.168.1.2 # storage network
option remote-subvolume ns1
# option transport-timeout 10 # value in seconds; it should
be set relatively low
end-volume
# automatic file replication translator for dataspace
volume afr1
type cluster/afr
subvolumes ds-threads1 ds3 # local and remote dataspaces
end-volume
# automatic file replication translator for dataspace
volume afr2
type cluster/afr
subvolumes ds-threads2 ds4 # local and remote dataspaces
end-volume
# automatic file replication translator for dataspace
volume afr-ns
type cluster/afr
subvolumes ns1 ns2 # local and remote dataspaces
end-volume
volume unify
type cluster/unify
subvolumes afr1 afr2
option namespace afr-ns # should be a node which is not present in
'subvolumes'
option scheduler rr # simple round-robin scheduler
end-volume
volume writebehind
type performance/write-behind
option aggregate-size 128kB
subvolumes unify
end-volume
volume readahead
type performance/read-ahead
option page-size 64kB
option page-count 16
subvolumes writebehind
end-volume
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080926/0d3a9612/attachment.html>