Gluster users - Apr 2018 - Problem adding replicated bricks on FreeBSD

Hi Folks,

I'm trying to debug an issue that I've found while attempting to qualify
GlusterFS for potential distributed storage projects on the FreeBSD-11.1  
server platform - using the existing package of GlusterFS v3.11.1_4

The main issue I've encountered is that I cannot add new bricks while  
setting/increasing the replica count.

If I create a replicated volume "poc" on two hosts, say
s1:/gluster/1/poc
and s2:/gluster/1/poc, the volume is created properly and shows replicated  
status, files are written to both volumes.

If I create a single volume: s1:/gluster/1/poc as a single / distributed  
brick, and then try to run

gluster volume add-brick poc replica 2 s2:/gluster/1/poc

it will always fail (sometimes after a pause, sometimes not.)  The only  
error I'm seeing on the server hosting the new brick, aside from the  
generic "Unable to add bricks" message, is like so:

I [MSGID: 106578]  
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:  
replica-count is set 2
I [MSGID: 106578]  
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:  
type is set 2, need to change it
E [MSGID: 106054]  
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:  
Failed to set extended attribute trusted.add-brick : Operation not  
supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]  
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:  
Add-brick commit failed.

I was initially using ZFS and noted that ZFS on FreeBSD does not support  
xattr, so I reverted to using UFS as the storage type for the brick, and  
still encounter this behavior.

I also recompiled the port (again, GlusterFS v3.11.1) with the patch from  
https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals  
specifically with xattr handling in FreeBSD.

To recap - I'm able to create any type of volume (2 or 3-way replicated or  
distributed), but I'm unable to add replicated bricks to a volume.

I was, however, able to add a second distributed brick ( gluster volume  
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding  
and/or changing the replica count while adding a new brick.

Please let me know if there are any other issues in addition to bug  
#1452961 I should be aware of, or additional log or debug info I can  
provide.

Best Regards,
Mark

On Thu, Apr 26, 2018 at 9:06 PM Mark Staudinger <mark.staudinger at
nyi.net>
wrote:
> Hi Folks,
> I'm trying to debug an issue that I've found while attempting to
qualify
> GlusterFS for potential distributed storage projects on the FreeBSD-11.1
> server platform - using the existing package of GlusterFS v3.11.1_4
> The main issue I've encountered is that I cannot add new bricks while
> setting/increasing the replica count.
> If I create a replicated volume "poc" on two hosts, say
s1:/gluster/1/poc
> and s2:/gluster/1/poc, the volume is created properly and shows replicated
> status, files are written to both volumes.
> If I create a single volume: s1:/gluster/1/poc as a single / distributed
> brick, and then try to run
> gluster volume add-brick poc replica 2 s2:/gluster/1/poc
> it will always fail (sometimes after a pause, sometimes not.)  The only
> error I'm seeing on the server hosting the new brick, aside from the
> generic "Unable to add bricks" message, is like so:
> I [MSGID: 106578]
> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
> replica-count is set 2
> I [MSGID: 106578]
> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
> type is set 2, need to change it
> E [MSGID: 106054]
> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:
> Failed to set extended attribute trusted.add-brick : Operation not
> supported [Operation not supported]
The log here seems to indicate the filesystem on the new brick being added
doesn't seem to support setting xattrs.
Maybe check the new brick again?
> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
> 0-glusterd: Unable to add bricks
> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:
> Add-brick commit failed.
> I was initially using ZFS and noted that ZFS on FreeBSD does not support
> xattr, so I reverted to using UFS as the storage type for the brick, and
> still encounter this behavior.
> I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
> https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
> specifically with xattr handling in FreeBSD.
> To recap - I'm able to create any type of volume (2 or 3-way replicated
or
> distributed), but I'm unable to add replicated bricks to a volume.
> I was, however, able to add a second distributed brick ( gluster volume
> add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
> and/or changing the replica count while adding a new brick.
> Please let me know if there are any other issues in addition to bug
> #1452961 I should be aware of, or additional log or debug info I can
> provide.
> Best Regards,
> Mark
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

Kaushal M wrote:
> On Thu, Apr 26, 2018 at 9:06 PM Mark Staudinger <mark.staudinger at
nyi.net>
> wrote:
>
>> Hi Folks,
>> I'm trying to debug an issue that I've found while attempting
to qualify
>> GlusterFS for potential distributed storage projects on the
FreeBSD-11.1
>> server platform - using the existing package of GlusterFS v3.11.1_4
>> The main issue I've encountered is that I cannot add new bricks
while
>> setting/increasing the replica count.
>> If I create a replicated volume "poc" on two hosts, say
s1:/gluster/1/poc
>> and s2:/gluster/1/poc, the volume is created properly and shows
replicated
>> status, files are written to both volumes.
>> If I create a single volume: s1:/gluster/1/poc as a single /
distributed
>> brick, and then try to run
>> gluster volume add-brick poc replica 2 s2:/gluster/1/poc
>> it will always fail (sometimes after a pause, sometimes not.)  The only
>> error I'm seeing on the server hosting the new brick, aside from
the
>> generic "Unable to add bricks" message, is like so:
>> I [MSGID: 106578]
>> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks]
0-management:
>> replica-count is set 2
>> I [MSGID: 106578]
>> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks]
0-management:
>> type is set 2, need to change it
>> E [MSGID: 106054]
>> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops]
0-management:
>> Failed to set extended attribute trusted.add-brick : Operation not
>> supported [Operation not supported]
> The log here seems to indicate the filesystem on the new brick being added
> doesn't seem to support setting xattrs.
> Maybe check the new brick again?
>
>> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
>> 0-glusterd: Unable to add bricks
>> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn]
0-management:
>> Add-brick commit failed.
>> I was initially using ZFS and noted that ZFS on FreeBSD does not
support
>> xattr, so I reverted to using UFS as the storage type for the brick,
and
>> still encounter this behavior.
>> I also recompiled the port (again, GlusterFS v3.11.1) with the patch
from
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
>> specifically with xattr handling in FreeBSD.
>> To recap - I'm able to create any type of volume (2 or 3-way
replicated or
>> distributed), but I'm unable to add replicated bricks to a volume.
>> I was, however, able to add a second distributed brick ( gluster volume
>> add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to
adding
>> and/or changing the replica count while adding a new brick.
>> Please let me know if there are any other issues in addition to bug
>> #1452961 I should be aware of, or additional log or debug info I can
>> provide.
>> Best Regards,
>> Mark
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
Hi Kaushal,

That's what the error message would indicate, but UFS2 does support 
xattr so that's a bit unexpected. Since I'm able to add a distributed 
brick, I can rule out basic setup issues.

All the same, I can see via 'truss'

extattr_set_link(0x8081ff890,0x1,0x803b27886,0x805258c40,0x400) ERR#45 
'Operation not supported?

Is anyone else using a similar setup (FreeBSD >= 11.1-RELEASE with the 
official pkg) that has tested adding bricks in this way? Are there any 
changes to the newfs or mount settings required for this to work that 
aren't obvious?

Cheers,
-=Mark

Mark Staudinger wrote:
> Hi Folks,
>
> I'm trying to debug an issue that I've found while attempting to 
> qualify GlusterFS for potential distributed storage projects on the 
> FreeBSD-11.1 server platform - using the existing package of GlusterFS 
> v3.11.1_4
>
> The main issue I've encountered is that I cannot add new bricks while 
> setting/increasing the replica count.
>
> If I create a replicated volume "poc" on two hosts, say 
> s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created 
> properly and shows replicated status, files are written to both volumes.
>
> If I create a single volume: s1:/gluster/1/poc as a single / 
> distributed brick, and then try to run
>
> gluster volume add-brick poc replica 2 s2:/gluster/1/poc
>
> it will always fail (sometimes after a pause, sometimes not.)? The 
> only error I'm seeing on the server hosting the new brick, aside from 
> the generic "Unable to add bricks" message, is like so:
>
> I [MSGID: 106578] 
> [glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 
> 0-management: replica-count is set 2
> I [MSGID: 106578] 
> [glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 
> 0-management: type is set 2, need to change it
> E [MSGID: 106054] 
> [glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
> 0-management: Failed to set extended attribute trusted.add-brick : 
> Operation not supported [Operation not supported]
> E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick] 
> 0-glusterd: Unable to add bricks
> E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 
> 0-management: Add-brick commit failed.
>
> I was initially using ZFS and noted that ZFS on FreeBSD does not 
> support xattr, so I reverted to using UFS as the storage type for the 
> brick, and still encounter this behavior.
>
> I also recompiled the port (again, GlusterFS v3.11.1) with the patch 
> from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals 
> specifically with xattr handling in FreeBSD.
>
> To recap - I'm able to create any type of volume (2 or 3-way 
> replicated or distributed), but I'm unable to add replicated bricks to 
> a volume.
>
> I was, however, able to add a second distributed brick ( gluster 
> volume add-brick poc s2:/gluster/1/poc ) - so the issue seems specific 
> to adding and/or changing the replica count while adding a new brick.
>
> Please let me know if there are any other issues in addition to bug 
> #1452961 I should be aware of, or additional log or debug info I can 
> provide.
>
>
Hi Folks,

As a follow-up to this, and it not making sense that xattrs would not be 
supported on UFS2, I am attaching the 'truss' output of the extattr 
syscalls from two operations.? In both situations, the command was 
initiated on server1, and traced on server2.

The first is a working "create-brick" issued from server1, traced from
server2: ( gluster volume create poc transport tcp replica 2 
s1:/gluster/1/poc s2:/gluster/1/poc )

extattr_set_link(0x807a55440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x807a55440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87 
'Attribute not found'
extattr_set_link(0x807a55440,0x1,0x803b24c17,0x807dfe250,0x10) = 16 (0x10)


And the second is from the not-working situation where the brick from s2 
is being added to the volume ( gluster volume add-brick poc replica 2 
s2:/gluster/1/poc commit )

extattr_set_link(0x80545c440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x80545c440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87 
'Attribute not found'
extattr_set_link(0x80545c440,0x1,0x803b24c17,0x8051e9c20,0x10) = 16 (0x10)
extattr_set_link(0x80545c440,0x1,0x803b2774c,0x807bff068,0xc) = 12 (0xc)
write(6," 31:???? option afr-pending-xattr 
poc-client-0,poc-client-1\n",64) = 64 (0x40)
extattr_set_link(0x807bff890,0x1,0x803b27886,0x80545d440,0x400) ERR#45 
'Operation not supported'
write(6,"[2018-04-28 15:21:51.271551] E [MSGID: 106054] 
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 
0-management: Failed to set extended attribute trusted.add-brick : 
Operation not supported [Operation not supported]\n",225) = 225 (0xe1)

Cheers,
-=Mark