[Gluster-users] fix-layout stalls with xattr errors

Dan Bretherton
Thu, 29 Dec 2011 00:35:37 -0000

Hello All-
I am having problems with rebalance ... fix-layout in version 3.2.5.  I
extended a volume with add-brick but the fix-layout stalls after a small
number of layout fixes and does not make any more progress.  I have
tried the operation twice on different servers with the same result.
  The following errors are found in the fuse mount log file on the
server carrying out the operation.

    [2011-12-28 21:38:14.840013] I
    [afr-common.c:1038:afr_launch_self_heal] 0-nemo2-replicate-4:
    background  data self-heal triggered. path:
    /users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.nc
    [2011-12-28 21:38:15.93079] E
    [client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-8:
    remote operation failed: Operation not supported
    [2011-12-28 21:38:15.93141] E
    [client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-9:
    remote operation failed: Operation not supported
    [2011-12-28 21:38:15.93385] I
    [client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-8: remote
    operation failed: Operation not supported
    [2011-12-28 21:38:15.93521] I
    [client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-9: remote
    operation failed: Operation not supported


The file in the error message is a link, and it is not broken as seen
from the volume mount point or the bricks.

There are some worrying error messages in the brick log files for
nemo2-client-8 and nemo2-client-9.  Here are some exerpts from the
nemo2-client-8 log, which is similar to the 0-nemo2-client-9 log.

    [2011-12-28 21:23:05.827877] W [posix.c:3928:do_xattrop]
    0-nemo2-posix: Extended attributes not supported by filesystem
    [2011-12-28 21:23:05.827932] I
    [server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 8438:
    FXATTROP 0 (-2111276040) ==> -1 (Operation not support
    ed)
    [2011-12-28 21:23:05.828848] E [posix.c:4200:posix_fstat]
    0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
    supported
    [2011-12-28 21:23:05.828879] I
    [server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 8439: FSTAT
    0 (-2111276040) ==> -1 (Operation not supported)
    [2011-12-28 21:29:29.871213] W
    [socket.c:1494:__socket_proto_state_machine] 0-tcp.nemo2-server:
    reading from socket failed. Error (Transport endpoint i
    s not connected), peer (192.171.166.81:1003)
    [2011-12-28 21:29:29.871305] I
    [server-helpers.c:360:do_lock_table_cleanup] 0-nemo2-server: inodelk
    released on /users/hzu/DATA/ERAINT/ORCA025/2010/sno
    w_ERAINT_2010.nc
    [2011-12-28 21:29:29.871345] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on
    /users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.
    nc

    [2011-12-28 21:34:36.190023] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /
    [2011-12-28 21:34:36.190055] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users
    [2011-12-28 21:34:36.190086] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu
    [2011-12-28 21:34:36.190102] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA
    [2011-12-28 21:34:36.190135] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT
    [2011-12-28 21:34:36.190154] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025
    [2011-12-28 21:34:36.190171] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025/2009

      [2011-12-28 21:38:15.92433] I
    [server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 12228:
    FXATTROP 7 (-2111276040) ==> -1 (Operation not supported)
    [2011-12-28 21:38:15.92743] E [posix.c:4200:posix_fstat]
    0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
    supported
    [2011-12-28 21:38:15.92775] I
    [server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 12229:
    FSTAT 7 (-2111276040) ==> -1 (Operation not supported)


The backend filesystems are ext4 and the are mounted with options
"acl,user_xattr".  I tested extended attribute support (as suggested
here:
http://gluster.org/pipermail/gluster-users/2010-December/006257.html)
and could not find any problems, so I don't understand the "Extended
attributes not supported by filesystem" error.  The only unusual thing
about the filesystems is the reduced number of filesystem features
enabled compared to other bricks.  These are the ext4 features enabled.

has_journal ext_attr resize_inode dir_index filetype needs_recovery
sparse_super large_file

All the other bricks in the volume have these features plus extent,
flex_bg, huge_file, uninit_bg, dir_nlink and extra_isize.  I don't know
if any of these missing ext4 features are part of the problem.  Does
anybody know what's going on here?

Regards
Dan.


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111229/a5cecfb0/attachment.html>