Gluster users - Jun 2016 - gluster warning remote operation failed during recovery from backups

I'm restoring some data to gluster from TSM backups and the client errors
out trying to retrieve xattrs at some point during the restore, killing
progress:
...
Restoring       8,118,878
/storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_04.asc
[Done]
ANS1587W Unable to read extended attributes for object
/storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc
due to errno: 34, reason: Numerical result out of range
 ** Unsuccessful **
...

In the gluster fuse logs for the volume I see this:
[2016-06-16 10:07:55.622020] W [MSGID: 114031]
[client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-storage-client-2: remote
operation failed. Path:
/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc
(0e98a94b-7b86-4a72-88a9-a99a787e059d). Key: (null) [Numerical result out
of range]
[2016-06-16 10:07:55.622110] W [fuse-bridge.c:3353:fuse_xattr_cbk]
0-glusterfs-fuse: 76197165: GETXATTR((null))
/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc =>
-1 (Numerical result out of range)

I'm trying to understand if gluster is bubbling up errors to the TSM client
(gluster fault), or reporting errors the TSM client is generating (TSM
fault).

Thanks,
Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160616/5db312eb/attachment.html>

On Thu, Jun 16, 2016 at 3:05 PM, Steve Dainard <sdainard at spd1.com>
wrote:
> I'm restoring some data to gluster from TSM backups and the client
errors
> out trying to retrieve xattrs at some point during the restore, killing
> progress:
> ...
> Restoring       8,118,878
>
/storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_04.asc
> [Done]
> ANS1587W Unable to read extended attributes for object
>
/storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc
> due to errno: 34, reason: Numerical result out of range
>  ** Unsuccessful **
> ...
>
> In the gluster fuse logs for the volume I see this:
> [2016-06-16 10:07:55.622020] W [MSGID: 114031]
> [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-storage-client-2: remote
> operation failed. Path:
> /data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc
> (0e98a94b-7b86-4a72-88a9-a99a787e059d). Key: (null) [Numerical result out
of
> range]
> [2016-06-16 10:07:55.622110] W [fuse-bridge.c:3353:fuse_xattr_cbk]
> 0-glusterfs-fuse: 76197165: GETXATTR((null))
> /data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc
=>
> -1 (Numerical result out of range)
>
> I'm trying to understand if gluster is bubbling up errors to the TSM
client
> (gluster fault), or reporting errors the TSM client is generating (TSM
> fault).
>
Do you happen to see the same error reported by posix translator(s) in
any of the brick(s)? Doing that might help in figuring out where the
problem could be stemming from.

As per man (2) getxattr, ERANGE is seen when the size of the value
buffer is too small to hold the result. Would it be possible to strace
the TSM client and see the size of the value buffer being passed?
Also, doing an extended attribute dump of the file on the brick
directory (either through attr or getfattr) can help in determining
the size necessary to hold all attributes.

HTH,
Vijay